Jul 28 19:59:08.770996 ip-10-0-128-43 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 19:59:09.083172 ip-10-0-128-43 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 19:59:09.083172 ip-10-0-128-43 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 19:59:09.083172 ip-10-0-128-43 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 19:59:09.083172 ip-10-0-128-43 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 19:59:09.083172 ip-10-0-128-43 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 19:59:09.084610 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.084523    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 19:59:09.089928 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089905    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:59:09.089928 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089923    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:59:09.089928 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089927    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:59:09.089928 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089930    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:59:09.089928 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089933    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:59:09.089928 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089936    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089939    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089943    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089946    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089948    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089951    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089954    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089956    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089959    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089961    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089964    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089967    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089970    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089972    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089975    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089977    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089980    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089983    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089985    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089988    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:59:09.090133 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089991    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089993    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089996    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.089998    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090001    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090003    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090006    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090011    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090014    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090016    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090020    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090025    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090028    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090031    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090034    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090037    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090040    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090042    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090045    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:59:09.090739 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090047    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090050    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090052    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090054    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090057    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090059    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090062    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090064    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090067    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090069    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090072    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090074    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090076    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090079    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090082    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090085    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090087    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090090    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090092    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090095    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:59:09.091208 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090098    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090100    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090103    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090105    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090108    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090111    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090113    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090116    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090119    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090121    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090124    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090126    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090130    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090133    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090136    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090139    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090142    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090145    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090149    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:59:09.091693 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090152    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090155    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090158    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090160    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090163    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090165    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090168    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090170    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090173    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090175    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090178    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090180    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090183    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090186    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090190    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090193    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090881    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090888    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090891    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090895    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:59:09.092210 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090897    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090900    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090903    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090905    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090908    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090913    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090918    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090921    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090924    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090927    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090930    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090933    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090936    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090939    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090941    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090944    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090947    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090950    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090952    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:59:09.092728 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090955    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090957    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090960    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090963    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090965    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090968    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090970    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090973    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090977    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090981    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090984    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090986    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090990    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090993    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090996    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.090999    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091001    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091004    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091006    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:59:09.093206 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091009    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091012    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091014    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091017    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091019    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091021    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091024    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091026    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091029    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091031    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091034    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091036    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091039    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091041    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091043    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091046    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091048    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091051    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091055    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091059    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:59:09.093680 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091061    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091064    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091067    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091070    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091072    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091076    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091079    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091082    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091084    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091087    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091089    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091092    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091094    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091096    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091099    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091101    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091104    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091107    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091110    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091112    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:59:09.094185 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091114    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091117    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091120    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091122    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091124    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091127    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091129    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091132    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091134    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091136    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091140    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091142    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091145    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091148    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091151    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091155    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091158    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091513    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091518    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091520    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:59:09.094694 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091523    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091526    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091528    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091531    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091534    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091536    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091539    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091542    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091545    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091547    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091550    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091552    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091555    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091558    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091561    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091563    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091566    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091569    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091571    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091573    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:59:09.095198 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091576    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091578    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091584    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091587    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091590    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091592    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091594    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091597    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091599    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091603    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091606    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091609    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091613    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091615    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091618    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091621    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091623    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091626    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091628    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091631    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:59:09.095688 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091634    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091636    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091639    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091641    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091644    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091646    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091649    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091651    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091654    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091656    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091659    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091662    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091664    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091666    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091671    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091679    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091682    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091684    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091687    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:59:09.096184 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091689    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091692    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091696    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091699    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091714    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091717    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091720    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091722    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091725    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091728    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091730    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091733    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091736    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091738    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091740    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091743    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091745    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091748    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091751    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091753    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:59:09.096696 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091756    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091758    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091761    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091763    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091766    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091768    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091771    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091775    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091779    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091783    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091786    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091789    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091792    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091796    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091800    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091803    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.091806    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091871    2582 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091878    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 19:59:09.097213 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091885    2582 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091889    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091895    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091899    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091903    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091907    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091910    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091912    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091916    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091919    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091922    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091925    2582 flags.go:64] FLAG: --cgroup-root=""
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091928    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091931    2582 flags.go:64] FLAG: --client-ca-file=""
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091935    2582 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091938    2582 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091941    2582 flags.go:64] FLAG: --cluster-domain=""
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091944    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091947    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091950    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091954    2582 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091958    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091961    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 19:59:09.097674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091964    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091967    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091970    2582 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091973    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091978    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091981    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091984    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091988    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091991    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091994    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.091997    2582 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092000    2582 flags.go:64] FLAG: --enable-server="true"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092003    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092007    2582 flags.go:64] FLAG: --event-burst="100"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092010    2582 flags.go:64] FLAG: --event-qps="50"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092013    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092016    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092019    2582 flags.go:64] FLAG: --eviction-hard=""
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092022    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092025    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092028    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092031    2582 flags.go:64] FLAG: --eviction-soft=""
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092033    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092036    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092039    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 19:59:09.098247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092042    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092045    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092048    2582 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092051    2582 flags.go:64] FLAG: --feature-gates=""
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092055    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092058    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092061    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092064    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092067    2582 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092070    2582 flags.go:64] FLAG: --help="false"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092073    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-128-43.ec2.internal"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092077    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092080    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092083    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092086    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092090    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092092    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092095    2582 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092098    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092100    2582 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092103    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092106    2582 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092109    2582 flags.go:64] FLAG: --kube-reserved=""
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092112    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 19:59:09.099181 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092115    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092118    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092121    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092123    2582 flags.go:64] FLAG: --lock-file=""
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092126    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092130    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092133    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092138    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092141    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092144    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092146    2582 flags.go:64] FLAG: --logging-format="text"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092149    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092155    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092157    2582 flags.go:64] FLAG: --manifest-url=""
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092160    2582 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092164    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092167    2582 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092171    2582 flags.go:64] FLAG: --max-pods="110"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092174    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092178    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092181    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092184    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092187    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092190    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092193    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 19:59:09.100243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092199    2582 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092202    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092205    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092208    2582 flags.go:64] FLAG: --pod-cidr=""
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092211    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092217    2582 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092220    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092223    2582 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092226    2582 flags.go:64] FLAG: --port="10250"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092229    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092232    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e9b354e2c03f361f"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092235    2582 flags.go:64] FLAG: --qos-reserved=""
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092238    2582 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092241    2582 flags.go:64] FLAG: --register-node="true"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092244    2582 flags.go:64] FLAG: --register-with-taints=""
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092248    2582 flags.go:64] FLAG: --registry-burst="10"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092255    2582 flags.go:64] FLAG: --registry-qps="5"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092258    2582 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092261    2582 flags.go:64] FLAG: --reserved-memory=""
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092264    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092269    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092272    2582 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092275    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092278    2582 flags.go:64] FLAG: --runonce="false"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092280    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 19:59:09.101176 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092283    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092288    2582 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092291    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092293    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092296    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092300    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092302    2582 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092305    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092308    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092311    2582 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092313    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092316    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092327    2582 flags.go:64] FLAG: --system-cgroups=""
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092330    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092335    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092338    2582 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092340    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092344    2582 flags.go:64] FLAG: --tls-min-version=""
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092347    2582 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092349    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092352    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092355    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092358    2582 flags.go:64] FLAG: --v="2"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092362    2582 flags.go:64] FLAG: --version="false"
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092366    2582 flags.go:64] FLAG: --vmodule=""
Jul 28 19:59:09.102135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092370    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.092373    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092485    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092489    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092492    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092495    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092497    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092500    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092505    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092508    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092510    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092513    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092516    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092518    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092521    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092524    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092526    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092529    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092532    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092534    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:59:09.103135 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092537    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092540    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092542    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092545    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092547    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092550    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092552    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092555    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092557    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092561    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092565    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092567    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092571    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092574    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092578    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092581    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092583    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092586    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092589    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:59:09.103801 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092591    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092595    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092598    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092600    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092603    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092606    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092608    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092611    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092614    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092618    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092622    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092626    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092630    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092633    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092637    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092642    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092647    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092651    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092654    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:59:09.104598 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092657    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092659    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092662    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092665    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092667    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092670    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092673    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092676    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092680    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092683    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092686    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092688    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092691    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092693    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092698    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092716    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092720    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092725    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092728    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092731    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:59:09.105242 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092734    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092737    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092740    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092743    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092745    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092748    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092750    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092753    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092757    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092761    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092764    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092767    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092769    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092772    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092775    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092777    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092780    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092782    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092785    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:59:09.105883 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092787    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092792    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092794    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.092797    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.093321    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.100784    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.100802    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.101463    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.101483    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101572    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101580    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101586    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101591    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101596    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101601    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:59:09.106378 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101606    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101610    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101615    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101619    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101624    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101628    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101632    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101636    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101641    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101645    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101649    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101653    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101660    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101667    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101672    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101676    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101680    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101684    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101689    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101693    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:59:09.106822 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101697    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101717    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101721    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101728    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101732    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101737    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101741    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101745    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101750    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101754    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101758    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101762    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101766    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101771    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101776    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101780    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101784    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101788    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101793    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:59:09.107314 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101799    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101805    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101810    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101814    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101819    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101823    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101827    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101831    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101836    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101840    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101844    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101848    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101853    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101858    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101862    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101866    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101871    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101878    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101882    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101887    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:59:09.107858 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101891    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101896    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101900    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101905    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101909    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101913    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101918    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101922    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101926    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101930    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101935    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101939    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101943    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101947    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101951    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101956    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101960    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101964    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101968    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101973    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:59:09.108385 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101977    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101982    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101988    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101993    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.101998    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102003    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102007    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102011    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102017    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102021    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102026    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102031    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102035    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102039    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:59:09.109069 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.102048    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102233    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102242    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102248    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102252    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102257    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102261    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102265    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102271    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102277    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102282    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102286    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102290    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102295    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102299    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102303    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102307    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102311    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102316    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:59:09.109434 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102320    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102324    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102328    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102332    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102336    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102341    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102345    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102350    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102355    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102359    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102364    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102368    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102372    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102376    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102381    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102385    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102389    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102393    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102400    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102406    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:59:09.110144 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102411    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102415    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102420    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102424    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102429    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102433    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102438    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102442    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102446    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102450    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102454    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102458    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102462    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102467    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102471    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102475    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102479    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102484    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102488    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102493    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:59:09.110938 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102497    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102502    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102506    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102511    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102515    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102519    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102523    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102527    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102532    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102536    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102540    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102545    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102549    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102553    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102557    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102561    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102565    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102569    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102574    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:59:09.111754 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102578    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102582    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102586    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102590    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102594    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102599    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102603    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102607    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102611    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102615    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102619    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102624    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102628    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102632    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102637    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102641    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102645    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102650    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102655    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102659    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:59:09.112283 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102663    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:59:09.112790 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:09.102667    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:59:09.112790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.102675    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 19:59:09.112790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.102853    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 19:59:09.112790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.105427    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 19:59:09.112790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.106692    2582 server.go:1019] "Starting client certificate rotation"
Jul 28 19:59:09.112790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.106813    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 19:59:09.112790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.106843    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 19:59:09.112790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.109869    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 19:59:09.112790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.112630    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 19:59:09.122795 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.122774    2582 log.go:25] "Validated CRI v1 runtime API"
Jul 28 19:59:09.125167 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.125152    2582 log.go:25] "Validated CRI v1 image API"
Jul 28 19:59:09.125883 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.125870    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 19:59:09.128087 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.128060    2582 fs.go:135] Filesystem UUIDs: map[659862cb-913e-45f4-ad53-9fb34ed21983:/dev/nvme0n1p4 684fec84-aba8-4bb0-9f81-24f1b053db3a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 19:59:09.128136 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.128087    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 19:59:09.132773 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.132640    2582 manager.go:216] Machine: {Timestamp:2026-07-28 19:59:09.131639243 +0000 UTC m=+0.266537487 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099961 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d0494b54ad398a87d001bbffb8a38 SystemUUID:ec2d0494-b54a-d398-a87d-001bbffb8a38 BootID:83dd6a2d-f949-485b-9350-6e2f7fea2812 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:09:1d:09:7b:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:09:1d:09:7b:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:42:5a:79:74:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 19:59:09.132773 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.132768    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 19:59:09.132886 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.132841    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 19:59:09.133668 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.133643    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 19:59:09.133815 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.133670    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 19:59:09.134303 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.134293    2582 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 19:59:09.134335 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.134305    2582 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 19:59:09.134335 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.134312    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 19:59:09.134834 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.134824    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 19:59:09.136000 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.135987    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 19:59:09.136985 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.136969    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 19:59:09.139634 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.139620    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 19:59:09.139763 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.139753    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 19:59:09.139809 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.139783    2582 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 19:59:09.139809 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.139801    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 19:59:09.139868 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.139815    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 19:59:09.139868 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.139824    2582 kubelet.go:398] "Adding apiserver pod source"
Jul 28 19:59:09.139868 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.139833    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 19:59:09.140214 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.140201    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 19:59:09.140265 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.140220    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 19:59:09.142120 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.142101    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 19:59:09.143425 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.143412    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 19:59:09.143476 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.143441    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 19:59:09.144013 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.143999    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 19:59:09.144070 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144015    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 19:59:09.144070 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144021    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 19:59:09.144070 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144027    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 19:59:09.144070 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144035    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 19:59:09.144070 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144044    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 19:59:09.144070 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144056    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 19:59:09.144070 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144063    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 19:59:09.144070 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144070    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 19:59:09.144070 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144076    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 19:59:09.144451 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144084    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 19:59:09.144451 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144089    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 19:59:09.144680 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144670    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 19:59:09.144734 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.144687    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 19:59:09.148782 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.148763    2582 server.go:1268] "Started kubelet"
Jul 28 19:59:09.148873 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.148845    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 19:59:09.149559 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.149508    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 19:59:09.149643 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.149588    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 19:59:09.149786 ip-10-0-128-43 systemd[1]: Started Kubernetes Kubelet.
Jul 28 19:59:09.150948 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.150745    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 19:59:09.150948 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.150766    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 19:59:09.150948 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.150878    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 19:59:09.151265 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.151214    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 19:59:09.151783 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.151770    2582 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 19:59:09.155040 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.155022    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 19:59:09.156611 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.155883    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-43.ec2.internal.18c68c77f59da0c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-43.ec2.internal,UID:ip-10-0-128-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-43.ec2.internal,},FirstTimestamp:2026-07-28 19:59:09.148725447 +0000 UTC m=+0.283623692,LastTimestamp:2026-07-28 19:59:09.148725447 +0000 UTC m=+0.283623692,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-43.ec2.internal,}"
Jul 28 19:59:09.161071 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.161054    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 19:59:09.161137 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.161074    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 19:59:09.161190 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.161162    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:09.161754 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.161733    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 19:59:09.161754 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.161749    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 19:59:09.161902 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.161763    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 19:59:09.161902 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.161852    2582 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 19:59:09.161902 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.161881    2582 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 19:59:09.162105 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.162086    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6b8x"
Jul 28 19:59:09.162216 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.162199    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 19:59:09.162285 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.162219    2582 factory.go:55] Registering systemd factory
Jul 28 19:59:09.162285 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.162228    2582 factory.go:223] Registration of the systemd container factory successfully
Jul 28 19:59:09.162470 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.162446    2582 factory.go:153] Registering CRI-O factory
Jul 28 19:59:09.162470 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.162466    2582 factory.go:223] Registration of the crio container factory successfully
Jul 28 19:59:09.162537 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.162489    2582 factory.go:103] Registering Raw factory
Jul 28 19:59:09.162537 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.162502    2582 manager.go:1196] Started watching for new ooms in manager
Jul 28 19:59:09.162957 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.162941    2582 manager.go:349] Starting recovery of all containers
Jul 28 19:59:09.167393 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.167374    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6b8x"
Jul 28 19:59:09.172630 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.172604    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 19:59:09.172745 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.172721    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 19:59:09.173382 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.173368    2582 manager.go:354] Recovery completed
Jul 28 19:59:09.177929 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.177916    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:59:09.179622 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.179602    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:59:09.179746 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.179645    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:59:09.179746 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.179661    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:59:09.181149 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.181125    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 19:59:09.181149 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.181144    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 19:59:09.181290 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.181162    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 19:59:09.181290 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.181251    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:59:09.182009 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.181992    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:59:09.182009 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.182018    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:59:09.182144 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.182039    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:59:09.183177 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.183165    2582 policy_none.go:49] "None policy: Start"
Jul 28 19:59:09.183222 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.183180    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 19:59:09.183222 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.183192    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 19:59:09.185180 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.185169    2582 policy_none.go:47] "Start"
Jul 28 19:59:09.229121 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.228944    2582 manager.go:337] "Starting Device Plugin manager"
Jul 28 19:59:09.229228 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.229149    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 19:59:09.229228 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.229163    2582 server.go:87] "Starting device plugin registration server"
Jul 28 19:59:09.229423 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.229412    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 19:59:09.229473 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.229426    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 19:59:09.229551 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.229530    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 19:59:09.229646 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.229634    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 19:59:09.229716 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.229647    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 19:59:09.230242 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.230219    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 19:59:09.230317 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.230277    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:09.294547 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.294505    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 19:59:09.296648 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.295750    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 19:59:09.296648 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.295793    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 19:59:09.296648 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.295818    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 19:59:09.296648 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.295859    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 19:59:09.298319 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.298300    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:59:09.330267 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.330247    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:59:09.331174 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.331157    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:59:09.331265 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.331187    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:59:09.331265 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.331199    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:59:09.331265 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.331220    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-43.ec2.internal"
Jul 28 19:59:09.339339 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.339299    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-43.ec2.internal"
Jul 28 19:59:09.339339 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.339324    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-43.ec2.internal\": node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:09.354933 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.354909    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:09.396508 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.396481    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal"]
Jul 28 19:59:09.396629 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.396548    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:59:09.397447 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.397432    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:59:09.397527 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.397459    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:59:09.397527 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.397468    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:59:09.399756 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.399744    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:59:09.399868 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.399854    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:09.399925 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.399896    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:59:09.400443 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.400427    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:59:09.400515 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.400461    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:59:09.400515 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.400471    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:59:09.400515 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.400431    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:59:09.400621 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.400529    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:59:09.400621 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.400538    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:59:09.402737 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.402722    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:09.402831 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.402744    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:59:09.403768 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.403750    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:59:09.403843 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.403785    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:59:09.403843 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:09.403798    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:59:09.429366 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.429341    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-43.ec2.internal\" not found" node="ip-10-0-128-43.ec2.internal"
Jul 28 19:59:09.433645 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:09.433630    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-43.ec2.internal\" not found" node="ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.735580 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:10.735555    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:10.744186 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:10.744164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69f862974659170e7d3d3e56d742be3c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal\" (UID: \"69f862974659170e7d3d3e56d742be3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.744248 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:10.744193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c858bff10c5634a7a7b495e5272b508-config\") pod \"kube-apiserver-proxy-ip-10-0-128-43.ec2.internal\" (UID: \"7c858bff10c5634a7a7b495e5272b508\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.744248 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:10.744210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69f862974659170e7d3d3e56d742be3c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal\" (UID: \"69f862974659170e7d3d3e56d742be3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.835769 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:10.835737    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:10.845109 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:10.845088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c858bff10c5634a7a7b495e5272b508-config\") pod \"kube-apiserver-proxy-ip-10-0-128-43.ec2.internal\" (UID: \"7c858bff10c5634a7a7b495e5272b508\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.845202 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:10.845114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69f862974659170e7d3d3e56d742be3c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal\" (UID: \"69f862974659170e7d3d3e56d742be3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.845202 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:10.845130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69f862974659170e7d3d3e56d742be3c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal\" (UID: \"69f862974659170e7d3d3e56d742be3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.845202 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:10.845172    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69f862974659170e7d3d3e56d742be3c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal\" (UID: \"69f862974659170e7d3d3e56d742be3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.845345 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:10.845205    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69f862974659170e7d3d3e56d742be3c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal\" (UID: \"69f862974659170e7d3d3e56d742be3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.845345 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:10.845210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c858bff10c5634a7a7b495e5272b508-config\") pod \"kube-apiserver-proxy-ip-10-0-128-43.ec2.internal\" (UID: \"7c858bff10c5634a7a7b495e5272b508\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:10.936583 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:10.936511    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:11.011989 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.011954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:11.017381 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.017212    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:11.037403 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:11.037387    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:11.137943 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:11.137906    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:11.238491 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:11.238466    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:11.339003 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:11.338970    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:11.386533 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.386493    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 19:59:11.386982 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.386629    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 19:59:11.411730 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.411703    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:59:11.415676 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.415661    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:59:11.439101 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:11.439077    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:11.453509 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.453476    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 19:54:10 +0000 UTC" deadline="2028-02-26 05:01:55.06459256 +0000 UTC"
Jul 28 19:59:11.453509 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.453504    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13857h2m43.611092112s"
Jul 28 19:59:11.539876 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:11.539811    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:11.607768 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:11.607741    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c858bff10c5634a7a7b495e5272b508.slice/crio-3e665ce502f2dec3f6c8b892f760436279028a2c9d1c1256c44d352197b7d0aa WatchSource:0}: Error finding container 3e665ce502f2dec3f6c8b892f760436279028a2c9d1c1256c44d352197b7d0aa: Status 404 returned error can't find the container with id 3e665ce502f2dec3f6c8b892f760436279028a2c9d1c1256c44d352197b7d0aa
Jul 28 19:59:11.608154 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:11.608140    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f862974659170e7d3d3e56d742be3c.slice/crio-a39e30667844c2d6df94139362bc5f9cea52430634d0505a3f161b0c9c7c023b WatchSource:0}: Error finding container a39e30667844c2d6df94139362bc5f9cea52430634d0505a3f161b0c9c7c023b: Status 404 returned error can't find the container with id a39e30667844c2d6df94139362bc5f9cea52430634d0505a3f161b0c9c7c023b
Jul 28 19:59:11.613908 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.613890    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 19:59:11.640530 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:11.640504    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-43.ec2.internal\" not found"
Jul 28 19:59:11.678590 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.678568    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:59:11.741885 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.741857    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:11.753001 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.752976    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 19:59:11.753947 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.753935    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal"
Jul 28 19:59:11.760063 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:11.760049    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 19:59:12.292833 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.292803    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:59:12.421636 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.421601    2582 apiserver.go:52] "Watching apiserver"
Jul 28 19:59:12.431589 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.431564    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 19:59:12.433033 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.433008    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-6bz87","openshift-cluster-node-tuning-operator/tuned-pvfwt","openshift-image-registry/node-ca-66np6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-s9lg5","kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th","openshift-dns/node-resolver-sfwvl","openshift-multus/multus-7n7lb","openshift-multus/multus-additional-cni-plugins-7jmcz","openshift-multus/network-metrics-daemon-nnc2m","openshift-network-diagnostics/network-check-target-fh5cb","openshift-network-operator/iptables-alerter-2rgjw","kube-system/global-pull-secret-syncer-pcwfl"]
Jul 28 19:59:12.435792 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.435768    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.437969 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.437925    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.438544 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.438513    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 19:59:12.438646 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.438583    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 19:59:12.438646 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.438617    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 19:59:12.438754 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.438707    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 19:59:12.438992 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.438818    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 19:59:12.438992 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.438953    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nnsqg\""
Jul 28 19:59:12.440209 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.440184    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:59:12.441346 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.440475    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 19:59:12.441346 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.440695    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.441346 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.441004    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 19:59:12.441505 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.441349    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h7694\""
Jul 28 19:59:12.443414 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.442253    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 19:59:12.443414 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.442792    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 19:59:12.443414 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.443134    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 19:59:12.443414 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.443186    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 19:59:12.443414 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.443388    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l7nw7\""
Jul 28 19:59:12.443656 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.443518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:12.443656 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.443586    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:12.445434 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.445141    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.450097 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.447647    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cbl52\""
Jul 28 19:59:12.450097 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.447873    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.450097 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.447943    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 19:59:12.450097 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.448025    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 19:59:12.450097 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.449074    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 19:59:12.450097 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.450074    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 19:59:12.450589 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.450565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.451314 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.450821    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 19:59:12.451314 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.450893    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lzwf8\""
Jul 28 19:59:12.451314 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.451038    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 19:59:12.451314 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.451140    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 19:59:12.451548 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.451336    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 19:59:12.452946 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.452924    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 19:59:12.453043 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.452964    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fcfqc\""
Jul 28 19:59:12.453217 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6899h\" (UniqueName: \"kubernetes.io/projected/294f6d80-9bc0-4a40-969a-3801b9c16c3e-kube-api-access-6899h\") pod \"node-ca-66np6\" (UID: \"294f6d80-9bc0-4a40-969a-3801b9c16c3e\") " pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.453284 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:12.453284 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-node-log\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.453284 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-env-overrides\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.453452 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453310    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-ovn-node-metrics-cert\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.453452 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr5bg\" (UniqueName: \"kubernetes.io/projected/6b9fc684-840f-4a31-8d0c-db439eab595a-kube-api-access-zr5bg\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.453452 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79130341-76cf-448d-9a66-7776250ed070-host-slash\") pod \"iptables-alerter-2rgjw\" (UID: \"79130341-76cf-448d-9a66-7776250ed070\") " pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.453452 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppwpd\" (UniqueName: \"kubernetes.io/projected/79130341-76cf-448d-9a66-7776250ed070-kube-api-access-ppwpd\") pod \"iptables-alerter-2rgjw\" (UID: \"79130341-76cf-448d-9a66-7776250ed070\") " pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.453452 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-slash\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.453690 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-etc-openvswitch\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.453690 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-cni-netd\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.453690 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xltlk\" (UniqueName: \"kubernetes.io/projected/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-kube-api-access-xltlk\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.453690 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b9fc684-840f-4a31-8d0c-db439eab595a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.453690 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/294f6d80-9bc0-4a40-969a-3801b9c16c3e-host\") pod \"node-ca-66np6\" (UID: \"294f6d80-9bc0-4a40-969a-3801b9c16c3e\") " pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.453690 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dedd29f8-8425-42cc-804d-2b786679bd20-dbus\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:12.453690 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-systemd-units\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.453690 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-run-openvswitch\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.453997 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.453767    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 19:59:12.454080 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454060    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.454302 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454370 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-ovnkube-script-lib\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454370 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.454460 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/294f6d80-9bc0-4a40-969a-3801b9c16c3e-serviceca\") pod \"node-ca-66np6\" (UID: \"294f6d80-9bc0-4a40-969a-3801b9c16c3e\") " pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.454460 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-kubelet\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454460 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454414    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-run-ovn\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454460 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-cni-bin\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454638 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-cnibin\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.454638 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dedd29f8-8425-42cc-804d-2b786679bd20-kubelet-config\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:12.454638 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-run-systemd\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454638 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-system-cni-dir\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.454638 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-os-release\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.454844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b9fc684-840f-4a31-8d0c-db439eab595a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.454844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-var-lib-openvswitch\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b9fc684-840f-4a31-8d0c-db439eab595a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.454844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79130341-76cf-448d-9a66-7776250ed070-iptables-alerter-script\") pod \"iptables-alerter-2rgjw\" (UID: \"79130341-76cf-448d-9a66-7776250ed070\") " pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.454844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-run-netns\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-log-socket\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.454844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.454829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-ovnkube-config\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.456402 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.456373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.456978 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.456954    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 19:59:12.457119 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.456953    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 19:59:12.457119 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.457055    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 19:59:12.457381 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.457362    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k6699\""
Jul 28 19:59:12.458704 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.458500    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:59:12.459991 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.459026    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlsrq\""
Jul 28 19:59:12.459991 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.459203    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 19:59:12.459991 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.459445    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:12.461537 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.461517    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:12.461621 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.461581    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:12.461898 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.461881    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 19:59:12.461979 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.461945    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s9j25\""
Jul 28 19:59:12.462137 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.462121    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 19:59:12.463537 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.463517    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:12.463607 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.463586    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:12.482824 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.482805    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fm9vr"
Jul 28 19:59:12.489862 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.489840    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fm9vr"
Jul 28 19:59:12.543737 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.543656    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 19:59:12.555572 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/efdf8223-ffee-4381-8f46-eb69be388114-multus-daemon-config\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.555723 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-run-multus-certs\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.555723 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b9fc684-840f-4a31-8d0c-db439eab595a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.555723 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dedd29f8-8425-42cc-804d-2b786679bd20-dbus\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:12.555723 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-run-openvswitch\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.555723 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.555723 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f789c89-5c77-4222-9acf-752bfb1b7994-tmp\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.556006 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.556006 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555754    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-run-openvswitch\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.556006 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.556139 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.556215 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dedd29f8-8425-42cc-804d-2b786679bd20-dbus\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:12.556349 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/294f6d80-9bc0-4a40-969a-3801b9c16c3e-serviceca\") pod \"node-ca-66np6\" (UID: \"294f6d80-9bc0-4a40-969a-3801b9c16c3e\") " pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.556349 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.555761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/294f6d80-9bc0-4a40-969a-3801b9c16c3e-serviceca\") pod \"node-ca-66np6\" (UID: \"294f6d80-9bc0-4a40-969a-3801b9c16c3e\") " pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.556349 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b9fc684-840f-4a31-8d0c-db439eab595a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.556349 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-kubelet\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.556349 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-cni-bin\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.556349 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-kubelet\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.556584 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-run-netns\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.556584 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-cni-bin\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.556584 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-kubernetes\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.556584 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-run-systemd\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.556584 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-os-release\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.556584 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-run-k8s-cni-cncf-io\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.556584 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556539    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-run-systemd\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.556584 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-var-lib-kubelet\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e3752e-cdc4-4616-b0ce-84df74553189-tmp-dir\") pod \"node-resolver-sfwvl\" (UID: \"17e3752e-cdc4-4616-b0ce-84df74553189\") " pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-sys-fs\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-system-cni-dir\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-os-release\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-system-cni-dir\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556756    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-os-release\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-multus-socket-dir-parent\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-registration-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.556912 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-device-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-etc-selinux\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-run-netns\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.556992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbt5\" (UniqueName: \"kubernetes.io/projected/6f23cdda-4df1-4276-8197-80943b7648b0-kube-api-access-qkbt5\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rzcs\" (UniqueName: \"kubernetes.io/projected/61fa245c-f0d0-44e9-8077-da9a73f46d35-kube-api-access-6rzcs\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-run-netns\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-log-socket\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-cnibin\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-log-socket\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-var-lib-cni-bin\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-system-cni-dir\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-hostroot\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppwpd\" (UniqueName: \"kubernetes.io/projected/79130341-76cf-448d-9a66-7776250ed070-kube-api-access-ppwpd\") pod \"iptables-alerter-2rgjw\" (UID: \"79130341-76cf-448d-9a66-7776250ed070\") " pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-slash\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557269 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-etc-openvswitch\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-cni-netd\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltlk\" (UniqueName: \"kubernetes.io/projected/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-kube-api-access-xltlk\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-slash\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-etc-openvswitch\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-modprobe-d\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/294f6d80-9bc0-4a40-969a-3801b9c16c3e-host\") pod \"node-ca-66np6\" (UID: \"294f6d80-9bc0-4a40-969a-3801b9c16c3e\") " pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-cni-netd\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-systemd-units\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-ovnkube-script-lib\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-multus-conf-dir\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/294f6d80-9bc0-4a40-969a-3801b9c16c3e-host\") pod \"node-ca-66np6\" (UID: \"294f6d80-9bc0-4a40-969a-3801b9c16c3e\") " pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-sysconfig\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-systemd-units\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-run-ovn\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-run-ovn\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec723a1d-bb69-4939-814a-23074980c3bb-konnectivity-ca\") pod \"konnectivity-agent-6bz87\" (UID: \"ec723a1d-bb69-4939-814a-23074980c3bb\") " pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-run\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.557897 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-host\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-cnibin\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b9fc684-840f-4a31-8d0c-db439eab595a-cnibin\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dedd29f8-8425-42cc-804d-2b786679bd20-kubelet-config\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efdf8223-ffee-4381-8f46-eb69be388114-cni-binary-copy\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557867    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthp7\" (UniqueName: \"kubernetes.io/projected/efdf8223-ffee-4381-8f46-eb69be388114-kube-api-access-hthp7\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwgk7\" (UniqueName: \"kubernetes.io/projected/9f789c89-5c77-4222-9acf-752bfb1b7994-kube-api-access-rwgk7\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dedd29f8-8425-42cc-804d-2b786679bd20-kubelet-config\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b9fc684-840f-4a31-8d0c-db439eab595a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-var-lib-openvswitch\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.557981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558009    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8v8\" (UniqueName: \"kubernetes.io/projected/17e3752e-cdc4-4616-b0ce-84df74553189-kube-api-access-vx8v8\") pod \"node-resolver-sfwvl\" (UID: \"17e3752e-cdc4-4616-b0ce-84df74553189\") " pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-socket-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-sys\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-ovnkube-script-lib\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-lib-modules\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-tuned\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.558749 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-var-lib-openvswitch\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b9fc684-840f-4a31-8d0c-db439eab595a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79130341-76cf-448d-9a66-7776250ed070-iptables-alerter-script\") pod \"iptables-alerter-2rgjw\" (UID: \"79130341-76cf-448d-9a66-7776250ed070\") " pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-sysctl-d\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-sysctl-conf\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-ovnkube-config\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-var-lib-cni-multus\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558331    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-systemd\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6899h\" (UniqueName: \"kubernetes.io/projected/294f6d80-9bc0-4a40-969a-3801b9c16c3e-kube-api-access-6899h\") pod \"node-ca-66np6\" (UID: \"294f6d80-9bc0-4a40-969a-3801b9c16c3e\") " pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b9fc684-840f-4a31-8d0c-db439eab595a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-node-log\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-env-overrides\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-ovn-node-metrics-cert\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-etc-kubernetes\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.559526 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-var-lib-kubelet\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr5bg\" (UniqueName: \"kubernetes.io/projected/6b9fc684-840f-4a31-8d0c-db439eab595a-kube-api-access-zr5bg\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79130341-76cf-448d-9a66-7776250ed070-host-slash\") pod \"iptables-alerter-2rgjw\" (UID: \"79130341-76cf-448d-9a66-7776250ed070\") " pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-multus-cni-dir\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec723a1d-bb69-4939-814a-23074980c3bb-agent-certs\") pod \"konnectivity-agent-6bz87\" (UID: \"ec723a1d-bb69-4939-814a-23074980c3bb\") " pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17e3752e-cdc4-4616-b0ce-84df74553189-hosts-file\") pod \"node-resolver-sfwvl\" (UID: \"17e3752e-cdc4-4616-b0ce-84df74553189\") " pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558756    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79130341-76cf-448d-9a66-7776250ed070-iptables-alerter-script\") pod \"iptables-alerter-2rgjw\" (UID: \"79130341-76cf-448d-9a66-7776250ed070\") " pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558816    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.558876    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-ovnkube-config\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.559000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79130341-76cf-448d-9a66-7776250ed070-host-slash\") pod \"iptables-alerter-2rgjw\" (UID: \"79130341-76cf-448d-9a66-7776250ed070\") " pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.559019    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.559093    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-env-overrides\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.559108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-node-log\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.559115    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret podName:dedd29f8-8425-42cc-804d-2b786679bd20 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:13.059078689 +0000 UTC m=+2.913389756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret") pod "global-pull-secret-syncer-pcwfl" (UID: "dedd29f8-8425-42cc-804d-2b786679bd20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.559138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b9fc684-840f-4a31-8d0c-db439eab595a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.560111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.559406    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 19:59:12.562633 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.562575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-ovn-node-metrics-cert\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.571637 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.571610    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltlk\" (UniqueName: \"kubernetes.io/projected/de3e2f3e-37ae-4b26-867e-d5e30f427b4b-kube-api-access-xltlk\") pod \"ovnkube-node-s9lg5\" (UID: \"de3e2f3e-37ae-4b26-867e-d5e30f427b4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.571736 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.571676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppwpd\" (UniqueName: \"kubernetes.io/projected/79130341-76cf-448d-9a66-7776250ed070-kube-api-access-ppwpd\") pod \"iptables-alerter-2rgjw\" (UID: \"79130341-76cf-448d-9a66-7776250ed070\") " pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.572598 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.572572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr5bg\" (UniqueName: \"kubernetes.io/projected/6b9fc684-840f-4a31-8d0c-db439eab595a-kube-api-access-zr5bg\") pod \"multus-additional-cni-plugins-7jmcz\" (UID: \"6b9fc684-840f-4a31-8d0c-db439eab595a\") " pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.573354 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.573335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6899h\" (UniqueName: \"kubernetes.io/projected/294f6d80-9bc0-4a40-969a-3801b9c16c3e-kube-api-access-6899h\") pod \"node-ca-66np6\" (UID: \"294f6d80-9bc0-4a40-969a-3801b9c16c3e\") " pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.580650 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.580601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal" event={"ID":"69f862974659170e7d3d3e56d742be3c","Type":"ContainerStarted","Data":"a39e30667844c2d6df94139362bc5f9cea52430634d0505a3f161b0c9c7c023b"}
Jul 28 19:59:12.581874 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.581854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal" event={"ID":"7c858bff10c5634a7a7b495e5272b508","Type":"ContainerStarted","Data":"3e665ce502f2dec3f6c8b892f760436279028a2c9d1c1256c44d352197b7d0aa"}
Jul 28 19:59:12.659247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659210    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-run-netns\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659247 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-kubernetes\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.659481 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-os-release\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659481 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-run-k8s-cni-cncf-io\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659481 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-var-lib-kubelet\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659481 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-run-netns\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659481 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e3752e-cdc4-4616-b0ce-84df74553189-tmp-dir\") pod \"node-resolver-sfwvl\" (UID: \"17e3752e-cdc4-4616-b0ce-84df74553189\") " pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.659481 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659365    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-kubernetes\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.659481 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-sys-fs\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.659481 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-multus-socket-dir-parent\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-registration-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-device-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-etc-selinux\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbt5\" (UniqueName: \"kubernetes.io/projected/6f23cdda-4df1-4276-8197-80943b7648b0-kube-api-access-qkbt5\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rzcs\" (UniqueName: \"kubernetes.io/projected/61fa245c-f0d0-44e9-8077-da9a73f46d35-kube-api-access-6rzcs\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-cnibin\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659629    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e3752e-cdc4-4616-b0ce-84df74553189-tmp-dir\") pod \"node-resolver-sfwvl\" (UID: \"17e3752e-cdc4-4616-b0ce-84df74553189\") " pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-var-lib-cni-bin\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-system-cni-dir\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-hostroot\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.659712 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-device-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-os-release\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-modprobe-d\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-sys-fs\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659752    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-run-k8s-cni-cncf-io\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-multus-conf-dir\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-multus-socket-dir-parent\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-multus-conf-dir\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-sysconfig\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659818    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec723a1d-bb69-4939-814a-23074980c3bb-konnectivity-ca\") pod \"konnectivity-agent-6bz87\" (UID: \"ec723a1d-bb69-4939-814a-23074980c3bb\") " pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-registration-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-run\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-etc-selinux\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659888    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-host\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efdf8223-ffee-4381-8f46-eb69be388114-cni-binary-copy\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthp7\" (UniqueName: \"kubernetes.io/projected/efdf8223-ffee-4381-8f46-eb69be388114-kube-api-access-hthp7\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwgk7\" (UniqueName: \"kubernetes.io/projected/9f789c89-5c77-4222-9acf-752bfb1b7994-kube-api-access-rwgk7\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660054 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8v8\" (UniqueName: \"kubernetes.io/projected/17e3752e-cdc4-4616-b0ce-84df74553189-kube-api-access-vx8v8\") pod \"node-resolver-sfwvl\" (UID: \"17e3752e-cdc4-4616-b0ce-84df74553189\") " pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.659985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-socket-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-sys\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-lib-modules\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-tuned\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-sysctl-d\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-sysctl-conf\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-var-lib-cni-multus\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-systemd\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-etc-kubernetes\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-var-lib-kubelet\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-multus-cni-dir\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec723a1d-bb69-4939-814a-23074980c3bb-agent-certs\") pod \"konnectivity-agent-6bz87\" (UID: \"ec723a1d-bb69-4939-814a-23074980c3bb\") " pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17e3752e-cdc4-4616-b0ce-84df74553189-hosts-file\") pod \"node-resolver-sfwvl\" (UID: \"17e3752e-cdc4-4616-b0ce-84df74553189\") " pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-cnibin\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/efdf8223-ffee-4381-8f46-eb69be388114-multus-daemon-config\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.660735 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660227    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec723a1d-bb69-4939-814a-23074980c3bb-konnectivity-ca\") pod \"konnectivity-agent-6bz87\" (UID: \"ec723a1d-bb69-4939-814a-23074980c3bb\") " pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-run-multus-certs\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-hostroot\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f789c89-5c77-4222-9acf-752bfb1b7994-tmp\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660272    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-run\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660283    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-var-lib-kubelet\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660412    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-sysconfig\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660412    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-multus-cni-dir\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-etc-kubernetes\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-var-lib-cni-bin\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660468    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-sys\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-var-lib-kubelet\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-lib-modules\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-system-cni-dir\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-var-lib-cni-multus\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660611    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-sysctl-d\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/efdf8223-ffee-4381-8f46-eb69be388114-multus-daemon-config\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.660991    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61fa245c-f0d0-44e9-8077-da9a73f46d35-socket-dir\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.661352 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.661092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/efdf8223-ffee-4381-8f46-eb69be388114-host-run-multus-certs\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.662050 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.661135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-systemd\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.662050 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.661159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-host\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.662050 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.661165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17e3752e-cdc4-4616-b0ce-84df74553189-hosts-file\") pod \"node-resolver-sfwvl\" (UID: \"17e3752e-cdc4-4616-b0ce-84df74553189\") " pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.662050 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.661241    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:12.662050 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.661255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-modprobe-d\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.662050 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.661306    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs podName:6f23cdda-4df1-4276-8197-80943b7648b0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:13.161274236 +0000 UTC m=+3.015585283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs") pod "network-metrics-daemon-nnc2m" (UID: "6f23cdda-4df1-4276-8197-80943b7648b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:12.662050 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.661384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-sysctl-conf\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.662050 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.661621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efdf8223-ffee-4381-8f46-eb69be388114-cni-binary-copy\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.663049 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.663027    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f789c89-5c77-4222-9acf-752bfb1b7994-etc-tuned\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.663155 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.663072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f789c89-5c77-4222-9acf-752bfb1b7994-tmp\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.663257 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.663237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec723a1d-bb69-4939-814a-23074980c3bb-agent-certs\") pod \"konnectivity-agent-6bz87\" (UID: \"ec723a1d-bb69-4939-814a-23074980c3bb\") " pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:12.668240 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.668219    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:59:12.669894 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.669866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwgk7\" (UniqueName: \"kubernetes.io/projected/9f789c89-5c77-4222-9acf-752bfb1b7994-kube-api-access-rwgk7\") pod \"tuned-pvfwt\" (UID: \"9f789c89-5c77-4222-9acf-752bfb1b7994\") " pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.669985 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.668245    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:59:12.670041 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.669995    2582 projected.go:196] Error preparing data for projected volume kube-api-access-jf4wn for pod openshift-network-diagnostics/network-check-target-fh5cb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:12.670041 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.670019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthp7\" (UniqueName: \"kubernetes.io/projected/efdf8223-ffee-4381-8f46-eb69be388114-kube-api-access-hthp7\") pod \"multus-7n7lb\" (UID: \"efdf8223-ffee-4381-8f46-eb69be388114\") " pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.670153 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.670056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbt5\" (UniqueName: \"kubernetes.io/projected/6f23cdda-4df1-4276-8197-80943b7648b0-kube-api-access-qkbt5\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:12.670153 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:12.670086    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn podName:b8f79db1-5f9e-4832-8647-08a142ef88e0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:13.170060133 +0000 UTC m=+3.024371181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jf4wn" (UniqueName: "kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn") pod "network-check-target-fh5cb" (UID: "b8f79db1-5f9e-4832-8647-08a142ef88e0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:12.670774 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.670750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rzcs\" (UniqueName: \"kubernetes.io/projected/61fa245c-f0d0-44e9-8077-da9a73f46d35-kube-api-access-6rzcs\") pod \"aws-ebs-csi-driver-node-6b4th\" (UID: \"61fa245c-f0d0-44e9-8077-da9a73f46d35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.673082 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.673062    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8v8\" (UniqueName: \"kubernetes.io/projected/17e3752e-cdc4-4616-b0ce-84df74553189-kube-api-access-vx8v8\") pod \"node-resolver-sfwvl\" (UID: \"17e3752e-cdc4-4616-b0ce-84df74553189\") " pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.748608 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.748563    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jmcz"
Jul 28 19:59:12.765104 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.765081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2rgjw"
Jul 28 19:59:12.772781 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.772755    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-66np6"
Jul 28 19:59:12.777571 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.777553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:12.784140 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.784121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sfwvl"
Jul 28 19:59:12.790683 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.790665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7n7lb"
Jul 28 19:59:12.797281 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.797237    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th"
Jul 28 19:59:12.803763 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.803745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvfwt"
Jul 28 19:59:12.809133 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:12.809112    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:13.062966 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.062896    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:13.063101 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:13.063008    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:13.063101 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:13.063062    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret podName:dedd29f8-8425-42cc-804d-2b786679bd20 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:14.063047027 +0000 UTC m=+3.917358072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret") pod "global-pull-secret-syncer-pcwfl" (UID: "dedd29f8-8425-42cc-804d-2b786679bd20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:13.163963 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.163927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:13.164121 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:13.164089    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:13.164173 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:13.164166    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs podName:6f23cdda-4df1-4276-8197-80943b7648b0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:14.164149904 +0000 UTC m=+4.018460948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs") pod "network-metrics-daemon-nnc2m" (UID: "6f23cdda-4df1-4276-8197-80943b7648b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:13.264736 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.264693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:13.264946 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:13.264896    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:59:13.264946 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:13.264932    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:59:13.264946 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:13.264947    2582 projected.go:196] Error preparing data for projected volume kube-api-access-jf4wn for pod openshift-network-diagnostics/network-check-target-fh5cb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:13.265130 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:13.265004    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn podName:b8f79db1-5f9e-4832-8647-08a142ef88e0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:14.264986006 +0000 UTC m=+4.119297049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jf4wn" (UniqueName: "kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn") pod "network-check-target-fh5cb" (UID: "b8f79db1-5f9e-4832-8647-08a142ef88e0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:13.408910 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:13.408869    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fa245c_f0d0_44e9_8077_da9a73f46d35.slice/crio-632af99609b2bc40c151113eba258a16d8380726dc0103928cf5113804e13153 WatchSource:0}: Error finding container 632af99609b2bc40c151113eba258a16d8380726dc0103928cf5113804e13153: Status 404 returned error can't find the container with id 632af99609b2bc40c151113eba258a16d8380726dc0103928cf5113804e13153
Jul 28 19:59:13.409426 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:13.409405    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294f6d80_9bc0_4a40_969a_3801b9c16c3e.slice/crio-5ee7ea2e9ec3614d3d965a514fbe3eade32a84fc91c28f1a9a9af795accb846f WatchSource:0}: Error finding container 5ee7ea2e9ec3614d3d965a514fbe3eade32a84fc91c28f1a9a9af795accb846f: Status 404 returned error can't find the container with id 5ee7ea2e9ec3614d3d965a514fbe3eade32a84fc91c28f1a9a9af795accb846f
Jul 28 19:59:13.410656 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:13.410617    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9fc684_840f_4a31_8d0c_db439eab595a.slice/crio-245138414c8ea447ee36bf8792530306e099f7fd9c299caa37b7573e7034011b WatchSource:0}: Error finding container 245138414c8ea447ee36bf8792530306e099f7fd9c299caa37b7573e7034011b: Status 404 returned error can't find the container with id 245138414c8ea447ee36bf8792530306e099f7fd9c299caa37b7573e7034011b
Jul 28 19:59:13.412018 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:13.411996    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f789c89_5c77_4222_9acf_752bfb1b7994.slice/crio-19e7091052c5ecee63b5fc949d8fe8898ec69bf9087472b4583bc11ffd5de23a WatchSource:0}: Error finding container 19e7091052c5ecee63b5fc949d8fe8898ec69bf9087472b4583bc11ffd5de23a: Status 404 returned error can't find the container with id 19e7091052c5ecee63b5fc949d8fe8898ec69bf9087472b4583bc11ffd5de23a
Jul 28 19:59:13.433730 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:13.433696    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79130341_76cf_448d_9a66_7776250ed070.slice/crio-acbc6abb7037d36d50618066b14bd174c694e2cc1b75b523a72758557620d871 WatchSource:0}: Error finding container acbc6abb7037d36d50618066b14bd174c694e2cc1b75b523a72758557620d871: Status 404 returned error can't find the container with id acbc6abb7037d36d50618066b14bd174c694e2cc1b75b523a72758557620d871
Jul 28 19:59:13.435250 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:13.435095    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec723a1d_bb69_4939_814a_23074980c3bb.slice/crio-d3cf0255a154bc442b32828c65bb8349378a8fb2b3eabff934e914e185a55e04 WatchSource:0}: Error finding container d3cf0255a154bc442b32828c65bb8349378a8fb2b3eabff934e914e185a55e04: Status 404 returned error can't find the container with id d3cf0255a154bc442b32828c65bb8349378a8fb2b3eabff934e914e185a55e04
Jul 28 19:59:13.436450 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:13.436346    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e3752e_cdc4_4616_b0ce_84df74553189.slice/crio-7135d053b818b1005000be1fdec022f2047d8ac068ca8234fe78e5ccaa9011f1 WatchSource:0}: Error finding container 7135d053b818b1005000be1fdec022f2047d8ac068ca8234fe78e5ccaa9011f1: Status 404 returned error can't find the container with id 7135d053b818b1005000be1fdec022f2047d8ac068ca8234fe78e5ccaa9011f1
Jul 28 19:59:13.437780 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:13.437700    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdf8223_ffee_4381_8f46_eb69be388114.slice/crio-24bb1a90625028f5f18c89146b1f8d26c55989729df36b71c82ae9cf2b4ce763 WatchSource:0}: Error finding container 24bb1a90625028f5f18c89146b1f8d26c55989729df36b71c82ae9cf2b4ce763: Status 404 returned error can't find the container with id 24bb1a90625028f5f18c89146b1f8d26c55989729df36b71c82ae9cf2b4ce763
Jul 28 19:59:13.438771 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:13.438753    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3e2f3e_37ae_4b26_867e_d5e30f427b4b.slice/crio-441b6fbec2a431011d8d9b128df0e3e82c83d7d8c254970ec1b7d5e254dd2405 WatchSource:0}: Error finding container 441b6fbec2a431011d8d9b128df0e3e82c83d7d8c254970ec1b7d5e254dd2405: Status 404 returned error can't find the container with id 441b6fbec2a431011d8d9b128df0e3e82c83d7d8c254970ec1b7d5e254dd2405
Jul 28 19:59:13.493260 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.493204    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:54:12 +0000 UTC" deadline="2028-02-10 15:16:23.581300013 +0000 UTC"
Jul 28 19:59:13.493260 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.493259    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13483h17m10.088044425s"
Jul 28 19:59:13.584495 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.584339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2rgjw" event={"ID":"79130341-76cf-448d-9a66-7776250ed070","Type":"ContainerStarted","Data":"acbc6abb7037d36d50618066b14bd174c694e2cc1b75b523a72758557620d871"}
Jul 28 19:59:13.585557 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.585527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvfwt" event={"ID":"9f789c89-5c77-4222-9acf-752bfb1b7994","Type":"ContainerStarted","Data":"19e7091052c5ecee63b5fc949d8fe8898ec69bf9087472b4583bc11ffd5de23a"}
Jul 28 19:59:13.586609 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.586583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerStarted","Data":"245138414c8ea447ee36bf8792530306e099f7fd9c299caa37b7573e7034011b"}
Jul 28 19:59:13.587728 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.587705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th" event={"ID":"61fa245c-f0d0-44e9-8077-da9a73f46d35","Type":"ContainerStarted","Data":"632af99609b2bc40c151113eba258a16d8380726dc0103928cf5113804e13153"}
Jul 28 19:59:13.588915 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.588889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7n7lb" event={"ID":"efdf8223-ffee-4381-8f46-eb69be388114","Type":"ContainerStarted","Data":"24bb1a90625028f5f18c89146b1f8d26c55989729df36b71c82ae9cf2b4ce763"}
Jul 28 19:59:13.589815 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.589799    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sfwvl" event={"ID":"17e3752e-cdc4-4616-b0ce-84df74553189","Type":"ContainerStarted","Data":"7135d053b818b1005000be1fdec022f2047d8ac068ca8234fe78e5ccaa9011f1"}
Jul 28 19:59:13.590708 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.590687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bz87" event={"ID":"ec723a1d-bb69-4939-814a-23074980c3bb","Type":"ContainerStarted","Data":"d3cf0255a154bc442b32828c65bb8349378a8fb2b3eabff934e914e185a55e04"}
Jul 28 19:59:13.593391 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.593363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-66np6" event={"ID":"294f6d80-9bc0-4a40-969a-3801b9c16c3e","Type":"ContainerStarted","Data":"5ee7ea2e9ec3614d3d965a514fbe3eade32a84fc91c28f1a9a9af795accb846f"}
Jul 28 19:59:13.595561 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.595537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal" event={"ID":"7c858bff10c5634a7a7b495e5272b508","Type":"ContainerStarted","Data":"3da726dca1ce82343ff59d3a1ff9e0a5cf924553c3428f7cb779c30c21aee41d"}
Jul 28 19:59:13.596901 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:13.596880    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" event={"ID":"de3e2f3e-37ae-4b26-867e-d5e30f427b4b","Type":"ContainerStarted","Data":"441b6fbec2a431011d8d9b128df0e3e82c83d7d8c254970ec1b7d5e254dd2405"}
Jul 28 19:59:14.070938 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.070903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:14.071114 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.071080    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:14.071174 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.071146    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret podName:dedd29f8-8425-42cc-804d-2b786679bd20 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:16.071127085 +0000 UTC m=+5.925438141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret") pod "global-pull-secret-syncer-pcwfl" (UID: "dedd29f8-8425-42cc-804d-2b786679bd20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:14.171597 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.171561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:14.171777 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.171713    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:14.171777 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.171775    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs podName:6f23cdda-4df1-4276-8197-80943b7648b0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:16.17175646 +0000 UTC m=+6.026067508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs") pod "network-metrics-daemon-nnc2m" (UID: "6f23cdda-4df1-4276-8197-80943b7648b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:14.272613 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.272568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:14.272797 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.272757    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:59:14.272797 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.272776    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:59:14.272797 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.272788    2582 projected.go:196] Error preparing data for projected volume kube-api-access-jf4wn for pod openshift-network-diagnostics/network-check-target-fh5cb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:14.272961 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.272855    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn podName:b8f79db1-5f9e-4832-8647-08a142ef88e0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:16.272825444 +0000 UTC m=+6.127136510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jf4wn" (UniqueName: "kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn") pod "network-check-target-fh5cb" (UID: "b8f79db1-5f9e-4832-8647-08a142ef88e0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:14.494488 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.494445    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:54:12 +0000 UTC" deadline="2028-02-26 09:02:08.051411797 +0000 UTC"
Jul 28 19:59:14.494488 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.494487    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13861h2m53.556929759s"
Jul 28 19:59:14.579516 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.579475    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:14.579686 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.579611    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:14.579748 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.579687    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:14.579832 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.579812    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:14.579899 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.579881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:14.579968 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:14.579950    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:14.627427 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.626680    2582 generic.go:356] "Generic (PLEG): container finished" podID="69f862974659170e7d3d3e56d742be3c" containerID="88e534d9f5e8177218ae4883bc74dff12ddf4b3b67e96da2fd39ea36b32c49ad" exitCode=0
Jul 28 19:59:14.627427 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.627363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal" event={"ID":"69f862974659170e7d3d3e56d742be3c","Type":"ContainerDied","Data":"88e534d9f5e8177218ae4883bc74dff12ddf4b3b67e96da2fd39ea36b32c49ad"}
Jul 28 19:59:14.660429 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:14.660375    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.ec2.internal" podStartSLOduration=3.660354828 podStartE2EDuration="3.660354828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:13.608840732 +0000 UTC m=+3.463151799" watchObservedRunningTime="2026-07-28 19:59:14.660354828 +0000 UTC m=+4.514665896"
Jul 28 19:59:15.670018 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:15.669843    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal" event={"ID":"69f862974659170e7d3d3e56d742be3c","Type":"ContainerStarted","Data":"d0d752372253db2be4a74ef667a288f8c9ca7f449a81bf7874eae0fcfe3b0f93"}
Jul 28 19:59:16.089743 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:16.089702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:16.089932 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.089853    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:16.089932 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.089911    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret podName:dedd29f8-8425-42cc-804d-2b786679bd20 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:20.08989214 +0000 UTC m=+9.944203189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret") pod "global-pull-secret-syncer-pcwfl" (UID: "dedd29f8-8425-42cc-804d-2b786679bd20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:16.190243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:16.190207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:16.190415 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.190393    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:16.190488 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.190456    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs podName:6f23cdda-4df1-4276-8197-80943b7648b0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:20.190437814 +0000 UTC m=+10.044748875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs") pod "network-metrics-daemon-nnc2m" (UID: "6f23cdda-4df1-4276-8197-80943b7648b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:16.292026 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:16.291414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:16.292026 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.291591    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:59:16.292026 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.291615    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:59:16.292026 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.291628    2582 projected.go:196] Error preparing data for projected volume kube-api-access-jf4wn for pod openshift-network-diagnostics/network-check-target-fh5cb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:16.292026 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.291688    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn podName:b8f79db1-5f9e-4832-8647-08a142ef88e0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:20.291670005 +0000 UTC m=+10.145981051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jf4wn" (UniqueName: "kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn") pod "network-check-target-fh5cb" (UID: "b8f79db1-5f9e-4832-8647-08a142ef88e0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:16.577233 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:16.577201    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:16.577233 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:16.577242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:16.577496 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:16.577201    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:16.577496 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.577346    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:16.577496 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.577443    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:16.577648 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:16.577525    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:18.577944 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:18.577706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:18.577944 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:18.577724    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:18.577944 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:18.577873    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:18.578532 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:18.578203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:18.578532 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:18.578322    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:18.578532 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:18.578395    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:20.126554 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:20.126453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:20.127009 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.126564    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:20.127009 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.126635    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret podName:dedd29f8-8425-42cc-804d-2b786679bd20 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:28.126616076 +0000 UTC m=+17.980927120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret") pod "global-pull-secret-syncer-pcwfl" (UID: "dedd29f8-8425-42cc-804d-2b786679bd20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:20.227819 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:20.227781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:20.227999 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.227969    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:20.228064 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.228033    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs podName:6f23cdda-4df1-4276-8197-80943b7648b0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:28.228013079 +0000 UTC m=+18.082324151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs") pod "network-metrics-daemon-nnc2m" (UID: "6f23cdda-4df1-4276-8197-80943b7648b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:20.328994 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:20.328944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:20.329182 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.329128    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:59:20.329182 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.329148    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:59:20.329182 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.329162    2582 projected.go:196] Error preparing data for projected volume kube-api-access-jf4wn for pod openshift-network-diagnostics/network-check-target-fh5cb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:20.329359 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.329222    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn podName:b8f79db1-5f9e-4832-8647-08a142ef88e0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:28.329203739 +0000 UTC m=+18.183514788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jf4wn" (UniqueName: "kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn") pod "network-check-target-fh5cb" (UID: "b8f79db1-5f9e-4832-8647-08a142ef88e0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:20.579421 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:20.578679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:20.579421 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.578789    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:20.579421 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:20.579151    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:20.579421 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.579248    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:20.579421 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:20.579318    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:20.579421 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:20.579389    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:22.577441 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:22.577404    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:22.577827 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:22.577540    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:22.577827 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:22.577738    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:22.577916 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:22.577829    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:22.577916 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:22.577897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:22.577995 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:22.577978    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:24.577499 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:24.577461    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:24.577499 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:24.577481    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:24.577953 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:24.577461    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:24.577953 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:24.577591    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:24.577953 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:24.577663    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:24.577953 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:24.577753    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:26.576780 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:26.576743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:26.577235 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:26.576743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:26.577235 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:26.576883    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:26.577235 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:26.577005    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:26.577235 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:26.577041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:26.577235 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:26.577104    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:28.190273 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:28.190238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:28.190711 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.190464    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:28.190711 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.190540    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret podName:dedd29f8-8425-42cc-804d-2b786679bd20 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:44.190518425 +0000 UTC m=+34.044829485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret") pod "global-pull-secret-syncer-pcwfl" (UID: "dedd29f8-8425-42cc-804d-2b786679bd20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:28.291388 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:28.291352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:28.291547 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.291505    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:28.291588 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.291569    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs podName:6f23cdda-4df1-4276-8197-80943b7648b0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:44.291554267 +0000 UTC m=+34.145865311 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs") pod "network-metrics-daemon-nnc2m" (UID: "6f23cdda-4df1-4276-8197-80943b7648b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:28.392412 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:28.392370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:28.392568 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.392527    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:59:28.392568 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.392551    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:59:28.392568 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.392564    2582 projected.go:196] Error preparing data for projected volume kube-api-access-jf4wn for pod openshift-network-diagnostics/network-check-target-fh5cb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:28.392693 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.392621    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn podName:b8f79db1-5f9e-4832-8647-08a142ef88e0 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:44.392602858 +0000 UTC m=+34.246913917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jf4wn" (UniqueName: "kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn") pod "network-check-target-fh5cb" (UID: "b8f79db1-5f9e-4832-8647-08a142ef88e0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:28.578137 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:28.577675    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:28.578137 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:28.577730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:28.578137 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:28.577676    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:28.578137 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.577813    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:28.578137 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.577925    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:28.578137 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:28.577986    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:30.577413 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:30.577381    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:30.577844 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:30.577485    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:30.577844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:30.577583    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:30.577844 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:30.577719    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:30.577844 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:30.577757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:30.578081 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:30.577847    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:32.577846 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.577440    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:32.578623 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.577524    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:32.578623 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:32.577950    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:32.578623 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.577532    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:32.578623 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:32.578047    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:32.578623 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:32.578101    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:32.707015 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.706983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" event={"ID":"de3e2f3e-37ae-4b26-867e-d5e30f427b4b","Type":"ContainerStarted","Data":"d57a550aa51a06c344c22ce0088b72218d03d2fb9bdf58a5fe3311cd4a9279c2"}
Jul 28 19:59:32.707142 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.707060    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" event={"ID":"de3e2f3e-37ae-4b26-867e-d5e30f427b4b","Type":"ContainerStarted","Data":"41a6642eca03ee2bf7c7db52d12d8fc1e5290ef0d22eb20944c55c43625ccf9d"}
Jul 28 19:59:32.707142 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.707101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" event={"ID":"de3e2f3e-37ae-4b26-867e-d5e30f427b4b","Type":"ContainerStarted","Data":"f378bbf6108c1782d7b7666181a92a3d89562bea585f730dbab18147f1d20881"}
Jul 28 19:59:32.707142 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.707132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" event={"ID":"de3e2f3e-37ae-4b26-867e-d5e30f427b4b","Type":"ContainerStarted","Data":"3596c75e82e76319a5b2ffda572ce9302aca50e2a4ed0d78ed323c961e490dfc"}
Jul 28 19:59:32.707274 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.707164    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" event={"ID":"de3e2f3e-37ae-4b26-867e-d5e30f427b4b","Type":"ContainerStarted","Data":"aa881af62fccf11fbe082e598d18d976e3ed66dffa1b8b447882b01336035cb6"}
Jul 28 19:59:32.707274 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.707195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" event={"ID":"de3e2f3e-37ae-4b26-867e-d5e30f427b4b","Type":"ContainerStarted","Data":"00193675784bce6a37488268ac77b7f17f3383ebd62b87061dc1fe16d6e206c9"}
Jul 28 19:59:32.708389 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.708365    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvfwt" event={"ID":"9f789c89-5c77-4222-9acf-752bfb1b7994","Type":"ContainerStarted","Data":"c51b14053a3c82ad67571936459a07f4a21cbb845e65de188fa91244aa91b9eb"}
Jul 28 19:59:32.709733 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.709712    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b9fc684-840f-4a31-8d0c-db439eab595a" containerID="760b72e2f25e354b461d54335c44d7cc8f4497e37b756bb4ce65db566c985f65" exitCode=0
Jul 28 19:59:32.709835 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.709786    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerDied","Data":"760b72e2f25e354b461d54335c44d7cc8f4497e37b756bb4ce65db566c985f65"}
Jul 28 19:59:32.711060 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.711041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th" event={"ID":"61fa245c-f0d0-44e9-8077-da9a73f46d35","Type":"ContainerStarted","Data":"ad7ecc4524f79cd0595fde79b12a332ce6557d5d769fc7327b9058025b9df386"}
Jul 28 19:59:32.712420 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.712401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7n7lb" event={"ID":"efdf8223-ffee-4381-8f46-eb69be388114","Type":"ContainerStarted","Data":"9222a916695ff4ecd90ac382d91e0dbe2da70913215a89ded70f0d880cb2bcf2"}
Jul 28 19:59:32.713682 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.713659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sfwvl" event={"ID":"17e3752e-cdc4-4616-b0ce-84df74553189","Type":"ContainerStarted","Data":"7a610b5b9a4e535d7d88c149ac7b52a1d19764bb839efe1873e065507c50eb5b"}
Jul 28 19:59:32.715002 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.714984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bz87" event={"ID":"ec723a1d-bb69-4939-814a-23074980c3bb","Type":"ContainerStarted","Data":"f0576f3583763b0399c38ef7afe2d6881accf26510f20c50e888b58848f319f3"}
Jul 28 19:59:32.716241 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.716224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-66np6" event={"ID":"294f6d80-9bc0-4a40-969a-3801b9c16c3e","Type":"ContainerStarted","Data":"381156e09b01908c4382ea4c9dc65b42a6fd7e8916f22d95da95c8311c0bce55"}
Jul 28 19:59:32.723020 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.722986    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pvfwt" podStartSLOduration=4.4967166 podStartE2EDuration="22.722974786s" totalImagesPullingTime="18.226258186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:32.722546941 +0000 UTC m=+22.576858022" watchObservedRunningTime="2026-07-28 19:59:32.722974786 +0000 UTC m=+22.577285852"
Jul 28 19:59:32.723111 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.723094    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.ec2.internal" podStartSLOduration=21.723090869 podStartE2EDuration="21.723090869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:15.68521713 +0000 UTC m=+5.539528197" watchObservedRunningTime="2026-07-28 19:59:32.723090869 +0000 UTC m=+22.577401935"
Jul 28 19:59:32.734050 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.734007    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sfwvl" podStartSLOduration=4.509046326 podStartE2EDuration="22.733992055s" totalImagesPullingTime="18.224945729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:32.733930861 +0000 UTC m=+22.588241928" watchObservedRunningTime="2026-07-28 19:59:32.733992055 +0000 UTC m=+22.588303125"
Jul 28 19:59:32.747304 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.747264    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6bz87" podStartSLOduration=4.522464554 podStartE2EDuration="22.747254515s" totalImagesPullingTime="18.224789961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:32.746778943 +0000 UTC m=+22.601090006" watchObservedRunningTime="2026-07-28 19:59:32.747254515 +0000 UTC m=+22.601565581"
Jul 28 19:59:32.778541 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.778492    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-66np6" podStartSLOduration=4.5307870359999995 podStartE2EDuration="22.778480927s" totalImagesPullingTime="18.247693891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:32.778146245 +0000 UTC m=+22.632457310" watchObservedRunningTime="2026-07-28 19:59:32.778480927 +0000 UTC m=+22.632791993"
Jul 28 19:59:32.794486 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.794451    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7n7lb" podStartSLOduration=4.5343573169999996 podStartE2EDuration="22.794440062s" totalImagesPullingTime="18.260082745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:32.794233811 +0000 UTC m=+22.648544877" watchObservedRunningTime="2026-07-28 19:59:32.794440062 +0000 UTC m=+22.648751186"
Jul 28 19:59:32.845714 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.845689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:32.916989 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:32.916962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:33.020229 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.020197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:33.104681 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.104657    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 19:59:33.215447 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.215416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:33.272992 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.272923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:33.374842 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.374779    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:33.478978 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.478942    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:33.522025 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.521915    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T19:59:33.104677569Z","UUID":"d9720c4b-73e8-46b3-871e-36dc99b2222d","Handler":null,"Name":"","Endpoint":""}
Jul 28 19:59:33.523608 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.523560    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 19:59:33.523608 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.523593    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 19:59:33.536139 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.536112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:33.720204 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.720167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2rgjw" event={"ID":"79130341-76cf-448d-9a66-7776250ed070","Type":"ContainerStarted","Data":"82f493b5d697d0a8d6c90334fc7f9fd33653146e1bc701dc1e1663f435cea34e"}
Jul 28 19:59:33.723018 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.722982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th" event={"ID":"61fa245c-f0d0-44e9-8077-da9a73f46d35","Type":"ContainerStarted","Data":"be2bfc2aec6832963d9042719706847a7ec367ac7992d2ab6baf86e33a179b36"}
Jul 28 19:59:33.735820 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.735777    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2rgjw" podStartSLOduration=5.509640371 podStartE2EDuration="23.735761511s" totalImagesPullingTime="18.22612114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:33.735160783 +0000 UTC m=+23.589471852" watchObservedRunningTime="2026-07-28 19:59:33.735761511 +0000 UTC m=+23.590072577"
Jul 28 19:59:33.864459 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.864376    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:33.915086 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:33.915054    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:34.257909 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:34.257864    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:34.359171 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:34.359143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:34.577102 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:34.577078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:34.577102 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:34.577100    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:34.577276 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:34.577194    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:34.577470 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:34.577450    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:34.577583 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:34.577557    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:34.577708 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:34.577677    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:34.729373 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:34.729332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" event={"ID":"de3e2f3e-37ae-4b26-867e-d5e30f427b4b","Type":"ContainerStarted","Data":"33e927285edc9a3e403094d94f0645b008e648c5f22526db6ce103b06b3b285e"}
Jul 28 19:59:34.731669 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:34.731641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th" event={"ID":"61fa245c-f0d0-44e9-8077-da9a73f46d35","Type":"ContainerStarted","Data":"e2d9d05a68d046634fcd646fc171a06573a553aafa53baf29fc9191836e9cb13"}
Jul 28 19:59:34.750740 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:34.750679    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b4th" podStartSLOduration=4.440423313 podStartE2EDuration="24.750660231s" totalImagesPullingTime="20.310236918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:34.750259833 +0000 UTC m=+24.604570925" watchObservedRunningTime="2026-07-28 19:59:34.750660231 +0000 UTC m=+24.604971298"
Jul 28 19:59:35.105612 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:35.105572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:35.160934 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:35.160900    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:35.460072 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:35.460000    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:35.517950 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:35.517909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:36.259187 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.259158    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:36.360847 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.360813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:36.577224 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.577149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:36.577224 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.577170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:36.577430 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:36.577269    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:36.577430 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:36.577354    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:36.577430 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.577406    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:36.577534 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:36.577478    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:36.637891 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.637859    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:36.638645 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.638620    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:36.672814 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.672782    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:36.734748 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.734719    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:36.735981 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.735947    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:36.736514 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.736493    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6bz87"
Jul 28 19:59:36.833253 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:36.833190    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:37.181373 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:37.181343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:37.289574 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:37.289545    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:37.740973 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:37.740940    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" event={"ID":"de3e2f3e-37ae-4b26-867e-d5e30f427b4b","Type":"ContainerStarted","Data":"5cdff43df8969e8ecca5bb6c1ea58eaf13195332399c019381dafceca7c148a6"}
Jul 28 19:59:37.741206 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:37.741179    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:37.746023 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:37.745995    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b9fc684-840f-4a31-8d0c-db439eab595a" containerID="6d01bcc671b1c6f7b9f714deca622b763f621c57c80681692c7ac5e107a4e6c3" exitCode=0
Jul 28 19:59:37.746139 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:37.746073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerDied","Data":"6d01bcc671b1c6f7b9f714deca622b763f621c57c80681692c7ac5e107a4e6c3"}
Jul 28 19:59:37.757093 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:37.756950    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:37.768356 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:37.768308    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5" podStartSLOduration=9.452828775 podStartE2EDuration="27.768276369s" totalImagesPullingTime="18.315447594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:37.768186954 +0000 UTC m=+27.622498021" watchObservedRunningTime="2026-07-28 19:59:37.768276369 +0000 UTC m=+27.622587436"
Jul 28 19:59:38.577405 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:38.577371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:38.577405 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:38.577368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:38.577928 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:38.577371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:38.577928 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:38.577512    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:38.577928 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:38.577586    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:38.577928 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:38.577625    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:38.750691 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:38.750657    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerStarted","Data":"2595a7fdbbcf459745b4a634d0777b55c4030b23b490a374e669a729f58aa1e1"}
Jul 28 19:59:38.751317 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:38.751279    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:38.751419 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:38.751351    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:38.765259 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:38.765240    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 19:59:39.114903 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.114821    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcwfl"]
Jul 28 19:59:39.115047 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.114956    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:39.115099 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:39.115039    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:39.117980 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.117957    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nnc2m"]
Jul 28 19:59:39.118113 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.118054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:39.118203 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:39.118144    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:39.118577 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.118555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fh5cb"]
Jul 28 19:59:39.118674 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.118636    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:39.118710 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:39.118697    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:39.243684 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.243655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:39.294209 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.294177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:39.754932 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.754893    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b9fc684-840f-4a31-8d0c-db439eab595a" containerID="2595a7fdbbcf459745b4a634d0777b55c4030b23b490a374e669a729f58aa1e1" exitCode=0
Jul 28 19:59:39.754932 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:39.754926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerDied","Data":"2595a7fdbbcf459745b4a634d0777b55c4030b23b490a374e669a729f58aa1e1"}
Jul 28 19:59:40.580008 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:40.579979    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:40.580152 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:40.579978    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:40.580152 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:40.580073    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:40.580152 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:40.579989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:40.580333 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:40.580135    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:40.580333 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:40.580259    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:41.761838 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:41.761674    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b9fc684-840f-4a31-8d0c-db439eab595a" containerID="096a3f771748129d4a2977ab5005a1ed542c6c934acc081f90310e7d4bc679f4" exitCode=0
Jul 28 19:59:41.761838 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:41.761761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerDied","Data":"096a3f771748129d4a2977ab5005a1ed542c6c934acc081f90310e7d4bc679f4"}
Jul 28 19:59:42.577584 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:42.577550    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:42.577767 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:42.577550    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:42.577767 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:42.577689    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:42.577767 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:42.577684    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh5cb" podUID="b8f79db1-5f9e-4832-8647-08a142ef88e0"
Jul 28 19:59:42.577946 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:42.577810    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcwfl" podUID="dedd29f8-8425-42cc-804d-2b786679bd20"
Jul 28 19:59:42.577946 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:42.577908    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nnc2m" podUID="6f23cdda-4df1-4276-8197-80943b7648b0"
Jul 28 19:59:44.208673 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.208633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:44.209228 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.208770    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:44.209228 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.208859    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret podName:dedd29f8-8425-42cc-804d-2b786679bd20 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:16.208833123 +0000 UTC m=+66.063144167 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret") pod "global-pull-secret-syncer-pcwfl" (UID: "dedd29f8-8425-42cc-804d-2b786679bd20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:59:44.302453 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.302419    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:44.309410 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.309385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:44.309555 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.309536    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:44.309624 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.309596    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs podName:6f23cdda-4df1-4276-8197-80943b7648b0 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:16.309580476 +0000 UTC m=+66.163891519 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs") pod "network-metrics-daemon-nnc2m" (UID: "6f23cdda-4df1-4276-8197-80943b7648b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:59:44.410387 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.410351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:44.410550 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.410498    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:59:44.410550 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.410515    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:59:44.410550 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.410528    2582 projected.go:196] Error preparing data for projected volume kube-api-access-jf4wn for pod openshift-network-diagnostics/network-check-target-fh5cb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:44.410729 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.410585    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn podName:b8f79db1-5f9e-4832-8647-08a142ef88e0 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:16.410568578 +0000 UTC m=+66.264879664 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jf4wn" (UniqueName: "kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn") pod "network-check-target-fh5cb" (UID: "b8f79db1-5f9e-4832-8647-08a142ef88e0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:59:44.481825 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.481745    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-43.ec2.internal" event="NodeReady"
Jul 28 19:59:44.481981 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.481873    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 19:59:44.514080 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.514051    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84844bb5c5-6bw2q"]
Jul 28 19:59:44.521692 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.521661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.524905 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.524881    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 19:59:44.525810 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.525786    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sp2jg"]
Jul 28 19:59:44.526153 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.526130    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 19:59:44.526153 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.526144    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6f6rv\""
Jul 28 19:59:44.526283 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.526133    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 19:59:44.529090 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.529072    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sm9vj"]
Jul 28 19:59:44.529175 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.529157    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.532386 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.532368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sm9vj"
Jul 28 19:59:44.532988 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.532966    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 19:59:44.533226 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.533207    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ph26b\""
Jul 28 19:59:44.533673 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.533240    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 19:59:44.533673 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.533207    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 19:59:44.534838 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.534819    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84844bb5c5-6bw2q"]
Jul 28 19:59:44.535243 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.535222    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 19:59:44.535479 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.535463    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 19:59:44.535572 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.535555    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 19:59:44.535572 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.535566    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pwmd4\""
Jul 28 19:59:44.542670 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.542649    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sm9vj"]
Jul 28 19:59:44.550398 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.550377    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sp2jg"]
Jul 28 19:59:44.578159 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.577386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 19:59:44.578159 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.577739    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 19:59:44.578159 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.577764    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 19:59:44.581034 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.580550    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 19:59:44.581034 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.580570    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 19:59:44.581034 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.580575    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kg9b\""
Jul 28 19:59:44.581034 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.580658    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 19:59:44.581034 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.580666    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 19:59:44.581034 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.580890    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gjwl7\""
Jul 28 19:59:44.611450 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c53d5b18-7246-472c-a974-2d2d0e161f80-ca-trust-extracted\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.611558 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-certificates\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.611558 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5339e904-5d69-4e23-bbc6-32bbda9e309d-config-volume\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.611648 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611551    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-bound-sa-token\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.611648 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvfp\" (UniqueName: \"kubernetes.io/projected/2da09d65-7fba-4191-ace4-dffe38a24a20-kube-api-access-tzvfp\") pod \"ingress-canary-sm9vj\" (UID: \"2da09d65-7fba-4191-ace4-dffe38a24a20\") " pod="openshift-ingress-canary/ingress-canary-sm9vj"
Jul 28 19:59:44.611648 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-installation-pull-secrets\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.611790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-trusted-ca\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.611790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.611790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5339e904-5d69-4e23-bbc6-32bbda9e309d-tmp-dir\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.611929 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnfdb\" (UniqueName: \"kubernetes.io/projected/5339e904-5d69-4e23-bbc6-32bbda9e309d-kube-api-access-hnfdb\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.611929 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2da09d65-7fba-4191-ace4-dffe38a24a20-cert\") pod \"ingress-canary-sm9vj\" (UID: \"2da09d65-7fba-4191-ace4-dffe38a24a20\") " pod="openshift-ingress-canary/ingress-canary-sm9vj"
Jul 28 19:59:44.611929 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kzx\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-kube-api-access-p7kzx\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.612040 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611958    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-image-registry-private-configuration\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.612040 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.611993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.695161 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.695126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:44.712784 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.712757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-certificates\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.712946 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.712801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5339e904-5d69-4e23-bbc6-32bbda9e309d-config-volume\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.712946 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.712824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-bound-sa-token\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.712946 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.712932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvfp\" (UniqueName: \"kubernetes.io/projected/2da09d65-7fba-4191-ace4-dffe38a24a20-kube-api-access-tzvfp\") pod \"ingress-canary-sm9vj\" (UID: \"2da09d65-7fba-4191-ace4-dffe38a24a20\") " pod="openshift-ingress-canary/ingress-canary-sm9vj"
Jul 28 19:59:44.713107 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.712962    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-installation-pull-secrets\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.713107 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.712994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-trusted-ca\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.713107 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.713107 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5339e904-5d69-4e23-bbc6-32bbda9e309d-tmp-dir\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.713107 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnfdb\" (UniqueName: \"kubernetes.io/projected/5339e904-5d69-4e23-bbc6-32bbda9e309d-kube-api-access-hnfdb\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.713376 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2da09d65-7fba-4191-ace4-dffe38a24a20-cert\") pod \"ingress-canary-sm9vj\" (UID: \"2da09d65-7fba-4191-ace4-dffe38a24a20\") " pod="openshift-ingress-canary/ingress-canary-sm9vj"
Jul 28 19:59:44.713426 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5339e904-5d69-4e23-bbc6-32bbda9e309d-tmp-dir\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.713474 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.713464    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:59:44.713539 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713518    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-certificates\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.713592 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.713525    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls podName:5339e904-5d69-4e23-bbc6-32bbda9e309d nodeName:}" failed. No retries permitted until 2026-07-28 19:59:45.213506467 +0000 UTC m=+35.067817516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls") pod "dns-default-sp2jg" (UID: "5339e904-5d69-4e23-bbc6-32bbda9e309d") : secret "dns-default-metrics-tls" not found
Jul 28 19:59:44.713592 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5339e904-5d69-4e23-bbc6-32bbda9e309d-config-volume\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.713707 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kzx\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-kube-api-access-p7kzx\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.713762 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-image-registry-private-configuration\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.713762 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.713853 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.713805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c53d5b18-7246-472c-a974-2d2d0e161f80-ca-trust-extracted\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.713966 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.713932    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:59:44.713966 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.713953    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84844bb5c5-6bw2q: secret "image-registry-tls" not found
Jul 28 19:59:44.714121 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:44.714003    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls podName:c53d5b18-7246-472c-a974-2d2d0e161f80 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:45.213985558 +0000 UTC m=+35.068296609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls") pod "image-registry-84844bb5c5-6bw2q" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80") : secret "image-registry-tls" not found
Jul 28 19:59:44.714121 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.714100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-trusted-ca\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.714219 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.714125    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c53d5b18-7246-472c-a974-2d2d0e161f80-ca-trust-extracted\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.717260 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.717240    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2da09d65-7fba-4191-ace4-dffe38a24a20-cert\") pod \"ingress-canary-sm9vj\" (UID: \"2da09d65-7fba-4191-ace4-dffe38a24a20\") " pod="openshift-ingress-canary/ingress-canary-sm9vj"
Jul 28 19:59:44.717260 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.717255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-installation-pull-secrets\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.717576 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.717557    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-image-registry-private-configuration\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.723316 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.723275    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-bound-sa-token\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.723418 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.723273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvfp\" (UniqueName: \"kubernetes.io/projected/2da09d65-7fba-4191-ace4-dffe38a24a20-kube-api-access-tzvfp\") pod \"ingress-canary-sm9vj\" (UID: \"2da09d65-7fba-4191-ace4-dffe38a24a20\") " pod="openshift-ingress-canary/ingress-canary-sm9vj"
Jul 28 19:59:44.724092 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.724070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnfdb\" (UniqueName: \"kubernetes.io/projected/5339e904-5d69-4e23-bbc6-32bbda9e309d-kube-api-access-hnfdb\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:44.725143 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.725121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kzx\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-kube-api-access-p7kzx\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:44.794603 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.794568    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:44.849979 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:44.849944    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sm9vj"
Jul 28 19:59:45.216439 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.216356    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:45.217717 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.217686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:45.217836 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.217736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:45.217907 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:45.217834    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:59:45.217907 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:45.217847    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:59:45.217907 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:45.217864    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84844bb5c5-6bw2q: secret "image-registry-tls" not found
Jul 28 19:59:45.217907 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:45.217889    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls podName:5339e904-5d69-4e23-bbc6-32bbda9e309d nodeName:}" failed. No retries permitted until 2026-07-28 19:59:46.217873254 +0000 UTC m=+36.072184314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls") pod "dns-default-sp2jg" (UID: "5339e904-5d69-4e23-bbc6-32bbda9e309d") : secret "dns-default-metrics-tls" not found
Jul 28 19:59:45.218087 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:45.217911    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls podName:c53d5b18-7246-472c-a974-2d2d0e161f80 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:46.217897162 +0000 UTC m=+36.072208207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls") pod "image-registry-84844bb5c5-6bw2q" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80") : secret "image-registry-tls" not found
Jul 28 19:59:45.319838 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.319803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:45.345862 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.345827    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gzjb6"]
Jul 28 19:59:45.349345 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.349326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.353083 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.353062    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 19:59:45.353316 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.353286    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 19:59:45.353316 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.353311    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 19:59:45.354478 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.354454    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mrvdk\""
Jul 28 19:59:45.354558 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.354456    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 19:59:45.360030 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.360006    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gzjb6"]
Jul 28 19:59:45.390010 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.389960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:45.520366 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.520334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec0121bc-2354-4424-9fd2-965012dcfc38-crio-socket\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.520516 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.520375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.520516 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.520421    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.520516 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.520468    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec0121bc-2354-4424-9fd2-965012dcfc38-data-volume\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.520655 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.520523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qhht\" (UniqueName: \"kubernetes.io/projected/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-api-access-6qhht\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.621914 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.621876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qhht\" (UniqueName: \"kubernetes.io/projected/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-api-access-6qhht\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.622170 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.621941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec0121bc-2354-4424-9fd2-965012dcfc38-crio-socket\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.622170 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.622112    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec0121bc-2354-4424-9fd2-965012dcfc38-crio-socket\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.622310 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.622186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.622310 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.622243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.622408 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.622332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec0121bc-2354-4424-9fd2-965012dcfc38-data-volume\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.622408 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:45.622349    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:59:45.622502 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:45.622424    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm podName:ec0121bc-2354-4424-9fd2-965012dcfc38 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:46.1224023 +0000 UTC m=+35.976713347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gzjb6" (UID: "ec0121bc-2354-4424-9fd2-965012dcfc38") : configmap "kube-rbac-proxy" not found
Jul 28 19:59:45.622502 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:45.622437    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:45.622607 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:45.622555    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls podName:ec0121bc-2354-4424-9fd2-965012dcfc38 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:46.122537628 +0000 UTC m=+35.976848686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzjb6" (UID: "ec0121bc-2354-4424-9fd2-965012dcfc38") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:45.622731 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.622712    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec0121bc-2354-4424-9fd2-965012dcfc38-data-volume\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.632625 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.632600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qhht\" (UniqueName: \"kubernetes.io/projected/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-api-access-6qhht\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:45.705339 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.705310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sm9vj"]
Jul 28 19:59:45.708674 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:45.708642    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da09d65_7fba_4191_ace4_dffe38a24a20.slice/crio-f283f81ce8db4404878d4397b6b2fe0744c534f37292de1267ea2a512421e8d4 WatchSource:0}: Error finding container f283f81ce8db4404878d4397b6b2fe0744c534f37292de1267ea2a512421e8d4: Status 404 returned error can't find the container with id f283f81ce8db4404878d4397b6b2fe0744c534f37292de1267ea2a512421e8d4
Jul 28 19:59:45.740101 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.740072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:45.775018 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.774913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerStarted","Data":"82b627d87866526369cbb6f1c7257c2bd26184cd027e5c470c6b99465a73bb13"}
Jul 28 19:59:45.776136 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.776113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sm9vj" event={"ID":"2da09d65-7fba-4191-ace4-dffe38a24a20","Type":"ContainerStarted","Data":"f283f81ce8db4404878d4397b6b2fe0744c534f37292de1267ea2a512421e8d4"}
Jul 28 19:59:45.802636 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.802610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:45.917934 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:45.917903    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:46.000599 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.000565    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:46.003529 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.003506    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm"]
Jul 28 19:59:46.005790 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.005776    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm"
Jul 28 19:59:46.008245 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.008227    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 19:59:46.008412 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.008307    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 19:59:46.008488 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.008472    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9jgfl\""
Jul 28 19:59:46.014819 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.014799    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm"]
Jul 28 19:59:46.064372 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.064310    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:46.117206 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.117178    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:46.127380 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.127358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:46.127454 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.127392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:46.127454 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.127428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2tvw\" (UniqueName: \"kubernetes.io/projected/2508efac-6c60-47b8-9656-9bf464750f3e-kube-api-access-k2tvw\") pod \"migrator-6865c9967-t47rm\" (UID: \"2508efac-6c60-47b8-9656-9bf464750f3e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm"
Jul 28 19:59:46.127540 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:46.127477    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:59:46.127540 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:46.127510    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:46.127540 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:46.127532    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm podName:ec0121bc-2354-4424-9fd2-965012dcfc38 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:47.127511935 +0000 UTC m=+36.981822979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gzjb6" (UID: "ec0121bc-2354-4424-9fd2-965012dcfc38") : configmap "kube-rbac-proxy" not found
Jul 28 19:59:46.127632 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:46.127549    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls podName:ec0121bc-2354-4424-9fd2-965012dcfc38 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:47.12753842 +0000 UTC m=+36.981849463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzjb6" (UID: "ec0121bc-2354-4424-9fd2-965012dcfc38") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:46.228393 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.228366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:46.228867 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.228439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:46.228867 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:46.228471    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:59:46.228867 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:46.228488    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84844bb5c5-6bw2q: secret "image-registry-tls" not found
Jul 28 19:59:46.228867 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.228531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2tvw\" (UniqueName: \"kubernetes.io/projected/2508efac-6c60-47b8-9656-9bf464750f3e-kube-api-access-k2tvw\") pod \"migrator-6865c9967-t47rm\" (UID: \"2508efac-6c60-47b8-9656-9bf464750f3e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm"
Jul 28 19:59:46.228867 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:46.228547    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls podName:c53d5b18-7246-472c-a974-2d2d0e161f80 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:48.228526462 +0000 UTC m=+38.082837524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls") pod "image-registry-84844bb5c5-6bw2q" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80") : secret "image-registry-tls" not found
Jul 28 19:59:46.228867 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:46.228564    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:59:46.228867 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:46.228602    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls podName:5339e904-5d69-4e23-bbc6-32bbda9e309d nodeName:}" failed. No retries permitted until 2026-07-28 19:59:48.228589836 +0000 UTC m=+38.082900884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls") pod "dns-default-sp2jg" (UID: "5339e904-5d69-4e23-bbc6-32bbda9e309d") : secret "dns-default-metrics-tls" not found
Jul 28 19:59:46.237515 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.237488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2tvw\" (UniqueName: \"kubernetes.io/projected/2508efac-6c60-47b8-9656-9bf464750f3e-kube-api-access-k2tvw\") pod \"migrator-6865c9967-t47rm\" (UID: \"2508efac-6c60-47b8-9656-9bf464750f3e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm"
Jul 28 19:59:46.315090 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.315004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm"
Jul 28 19:59:46.466411 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.466378    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm"]
Jul 28 19:59:46.469751 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:46.469726    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2508efac_6c60_47b8_9656_9bf464750f3e.slice/crio-ead149c8e9938566a75ff5b869bf7aea3d674a6ce331b157d45d2200033e5a6e WatchSource:0}: Error finding container ead149c8e9938566a75ff5b869bf7aea3d674a6ce331b157d45d2200033e5a6e: Status 404 returned error can't find the container with id ead149c8e9938566a75ff5b869bf7aea3d674a6ce331b157d45d2200033e5a6e
Jul 28 19:59:46.779561 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.779525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm" event={"ID":"2508efac-6c60-47b8-9656-9bf464750f3e","Type":"ContainerStarted","Data":"ead149c8e9938566a75ff5b869bf7aea3d674a6ce331b157d45d2200033e5a6e"}
Jul 28 19:59:46.782551 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.782528    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b9fc684-840f-4a31-8d0c-db439eab595a" containerID="82b627d87866526369cbb6f1c7257c2bd26184cd027e5c470c6b99465a73bb13" exitCode=0
Jul 28 19:59:46.782680 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:46.782573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerDied","Data":"82b627d87866526369cbb6f1c7257c2bd26184cd027e5c470c6b99465a73bb13"}
Jul 28 19:59:47.136666 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.136575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:47.136805 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.136743    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:47.136805 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:47.136745    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:47.136916 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:47.136780    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:59:47.136916 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:47.136864    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls podName:ec0121bc-2354-4424-9fd2-965012dcfc38 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:49.136837693 +0000 UTC m=+38.991148772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzjb6" (UID: "ec0121bc-2354-4424-9fd2-965012dcfc38") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:47.136916 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:47.136885    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm podName:ec0121bc-2354-4424-9fd2-965012dcfc38 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:49.136874738 +0000 UTC m=+38.991185788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gzjb6" (UID: "ec0121bc-2354-4424-9fd2-965012dcfc38") : configmap "kube-rbac-proxy" not found
Jul 28 19:59:47.154200 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.154167    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sfwvl_17e3752e-cdc4-4616-b0ce-84df74553189/dns-node-resolver/0.log"
Jul 28 19:59:47.515256 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.515026    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8s5w2"]
Jul 28 19:59:47.520076 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.520051    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.523137 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.522569    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 19:59:47.523137 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.522706    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 19:59:47.523137 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.522800    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 19:59:47.523137 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.523004    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 19:59:47.523857 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.523839    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4xnxl\""
Jul 28 19:59:47.527725 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.527704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8s5w2"]
Jul 28 19:59:47.552694 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.552663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:47.606971 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.606942    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:47.641259 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.641228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn2gk\" (UniqueName: \"kubernetes.io/projected/8bda1dd0-3f04-4e68-b64d-8878c5203ab7-kube-api-access-rn2gk\") pod \"service-ca-658d7b7c78-8s5w2\" (UID: \"8bda1dd0-3f04-4e68-b64d-8878c5203ab7\") " pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.641442 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.641271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8bda1dd0-3f04-4e68-b64d-8878c5203ab7-signing-key\") pod \"service-ca-658d7b7c78-8s5w2\" (UID: \"8bda1dd0-3f04-4e68-b64d-8878c5203ab7\") " pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.641508 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.641443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8bda1dd0-3f04-4e68-b64d-8878c5203ab7-signing-cabundle\") pod \"service-ca-658d7b7c78-8s5w2\" (UID: \"8bda1dd0-3f04-4e68-b64d-8878c5203ab7\") " pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.660599 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.660566    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:47.741921 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.741886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn2gk\" (UniqueName: \"kubernetes.io/projected/8bda1dd0-3f04-4e68-b64d-8878c5203ab7-kube-api-access-rn2gk\") pod \"service-ca-658d7b7c78-8s5w2\" (UID: \"8bda1dd0-3f04-4e68-b64d-8878c5203ab7\") " pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.742094 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.741935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8bda1dd0-3f04-4e68-b64d-8878c5203ab7-signing-key\") pod \"service-ca-658d7b7c78-8s5w2\" (UID: \"8bda1dd0-3f04-4e68-b64d-8878c5203ab7\") " pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.744784 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.744754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8bda1dd0-3f04-4e68-b64d-8878c5203ab7-signing-cabundle\") pod \"service-ca-658d7b7c78-8s5w2\" (UID: \"8bda1dd0-3f04-4e68-b64d-8878c5203ab7\") " pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.745508 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.745482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8bda1dd0-3f04-4e68-b64d-8878c5203ab7-signing-cabundle\") pod \"service-ca-658d7b7c78-8s5w2\" (UID: \"8bda1dd0-3f04-4e68-b64d-8878c5203ab7\") " pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.746910 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.746885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8bda1dd0-3f04-4e68-b64d-8878c5203ab7-signing-key\") pod \"service-ca-658d7b7c78-8s5w2\" (UID: \"8bda1dd0-3f04-4e68-b64d-8878c5203ab7\") " pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.754012 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.753987    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn2gk\" (UniqueName: \"kubernetes.io/projected/8bda1dd0-3f04-4e68-b64d-8878c5203ab7-kube-api-access-rn2gk\") pod \"service-ca-658d7b7c78-8s5w2\" (UID: \"8bda1dd0-3f04-4e68-b64d-8878c5203ab7\") " pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.789261 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.789182    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b9fc684-840f-4a31-8d0c-db439eab595a" containerID="03916fcadb62d3d0ccd1dea85e25fda8bc6b6011c4f32635a236fe7636edf413" exitCode=0
Jul 28 19:59:47.789261 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.789243    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerDied","Data":"03916fcadb62d3d0ccd1dea85e25fda8bc6b6011c4f32635a236fe7636edf413"}
Jul 28 19:59:47.832439 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.832407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2"
Jul 28 19:59:47.953666 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:47.953641    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-66np6_294f6d80-9bc0-4a40-969a-3801b9c16c3e/node-ca/0.log"
Jul 28 19:59:48.249488 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:48.249457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:48.249658 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:48.249514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:48.249658 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:48.249632    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:59:48.249658 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:48.249652    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84844bb5c5-6bw2q: secret "image-registry-tls" not found
Jul 28 19:59:48.249802 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:48.249722    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls podName:c53d5b18-7246-472c-a974-2d2d0e161f80 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:52.249698679 +0000 UTC m=+42.104009739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls") pod "image-registry-84844bb5c5-6bw2q" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80") : secret "image-registry-tls" not found
Jul 28 19:59:48.249802 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:48.249756    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:59:48.249897 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:48.249812    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls podName:5339e904-5d69-4e23-bbc6-32bbda9e309d nodeName:}" failed. No retries permitted until 2026-07-28 19:59:52.249798116 +0000 UTC m=+42.104109161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls") pod "dns-default-sp2jg" (UID: "5339e904-5d69-4e23-bbc6-32bbda9e309d") : secret "dns-default-metrics-tls" not found
Jul 28 19:59:48.378432 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:48.378363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:48.976577 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:48.976417    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8s5w2"]
Jul 28 19:59:48.979369 ip-10-0-128-43 kubenswrapper[2582]: W0728 19:59:48.979342    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bda1dd0_3f04_4e68_b64d_8878c5203ab7.slice/crio-cdf2550e60b69626dfc0b6c6cea229d5a8c7339e4933ee9ebf65eb7722d7be6d WatchSource:0}: Error finding container cdf2550e60b69626dfc0b6c6cea229d5a8c7339e4933ee9ebf65eb7722d7be6d: Status 404 returned error can't find the container with id cdf2550e60b69626dfc0b6c6cea229d5a8c7339e4933ee9ebf65eb7722d7be6d
Jul 28 19:59:49.158268 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.158238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:49.158429 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.158279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:49.158429 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:49.158398    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:49.158514 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:49.158458    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls podName:ec0121bc-2354-4424-9fd2-965012dcfc38 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:53.158442276 +0000 UTC m=+43.012753326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzjb6" (UID: "ec0121bc-2354-4424-9fd2-965012dcfc38") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:49.158848 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.158826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec0121bc-2354-4424-9fd2-965012dcfc38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:49.798814 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.798778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" event={"ID":"6b9fc684-840f-4a31-8d0c-db439eab595a","Type":"ContainerStarted","Data":"090b9b0cdd2272b03625291d49ad3ef12f2b657bfbb99e1af97fae6b5d8bdf56"}
Jul 28 19:59:49.800390 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.800360    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sm9vj" event={"ID":"2da09d65-7fba-4191-ace4-dffe38a24a20","Type":"ContainerStarted","Data":"08f036c72e68030c2da8656a0c79ab2ec7d24460ffbad0755dcacc9eaade27c9"}
Jul 28 19:59:49.802330 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.802286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm" event={"ID":"2508efac-6c60-47b8-9656-9bf464750f3e","Type":"ContainerStarted","Data":"76cff1edc826b3ea86fd6ea81afccf109e01418982b3a45b513dd6d8a02e3c5b"}
Jul 28 19:59:49.802434 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.802355    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm" event={"ID":"2508efac-6c60-47b8-9656-9bf464750f3e","Type":"ContainerStarted","Data":"8782c0183147026102c399f4308867b4a208fe7bb5074cc11b602fcdcb070dd2"}
Jul 28 19:59:49.804073 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.804036    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2" event={"ID":"8bda1dd0-3f04-4e68-b64d-8878c5203ab7","Type":"ContainerStarted","Data":"cdf2550e60b69626dfc0b6c6cea229d5a8c7339e4933ee9ebf65eb7722d7be6d"}
Jul 28 19:59:49.824210 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.824141    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7jmcz" podStartSLOduration=11.505381307 podStartE2EDuration="39.824128952s" totalImagesPullingTime="28.318747645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:49.822400086 +0000 UTC m=+39.676711161" watchObservedRunningTime="2026-07-28 19:59:49.824128952 +0000 UTC m=+39.678440017"
Jul 28 19:59:49.838905 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.838861    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sm9vj" podStartSLOduration=2.699236664 podStartE2EDuration="5.838847338s" totalImagesPullingTime="3.139610674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:49.838427712 +0000 UTC m=+39.692738791" watchObservedRunningTime="2026-07-28 19:59:49.838847338 +0000 UTC m=+39.693158406"
Jul 28 19:59:49.853497 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:49.853455    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-t47rm" podStartSLOduration=2.474546074 podStartE2EDuration="4.853445363s" totalImagesPullingTime="2.378899289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:49.852701061 +0000 UTC m=+39.707012126" watchObservedRunningTime="2026-07-28 19:59:49.853445363 +0000 UTC m=+39.707756428"
Jul 28 19:59:50.212116 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:50.212007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:50.914510 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:50.914427    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:51.811863 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:51.811827    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2" event={"ID":"8bda1dd0-3f04-4e68-b64d-8878c5203ab7","Type":"ContainerStarted","Data":"92e99a4d8ccb8e419a14862adf9f70a4b329508a9741a3083a679b11cd15afed"}
Jul 28 19:59:51.829708 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:51.829663    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-8s5w2" podStartSLOduration=2.404250771 podStartE2EDuration="4.829647545s" totalImagesPullingTime="2.425396774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:51.828612584 +0000 UTC m=+41.682923650" watchObservedRunningTime="2026-07-28 19:59:51.829647545 +0000 UTC m=+41.683958588"
Jul 28 19:59:52.282175 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:52.282143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 19:59:52.282378 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:52.282190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 19:59:52.282378 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:52.282277    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:59:52.282378 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:52.282286    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:59:52.282378 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:52.282328    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84844bb5c5-6bw2q: secret "image-registry-tls" not found
Jul 28 19:59:52.282378 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:52.282349    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls podName:5339e904-5d69-4e23-bbc6-32bbda9e309d nodeName:}" failed. No retries permitted until 2026-07-28 20:00:00.282335318 +0000 UTC m=+50.136646362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls") pod "dns-default-sp2jg" (UID: "5339e904-5d69-4e23-bbc6-32bbda9e309d") : secret "dns-default-metrics-tls" not found
Jul 28 19:59:52.282751 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:52.282393    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls podName:c53d5b18-7246-472c-a974-2d2d0e161f80 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:00.282380268 +0000 UTC m=+50.136691312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls") pod "image-registry-84844bb5c5-6bw2q" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80") : secret "image-registry-tls" not found
Jul 28 19:59:52.662356 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:52.662246    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:52.908970 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:52.904510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:53.003920 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:53.003859    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:53.104483 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:53.104451    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:53.191199 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:53.191165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 19:59:53.191658 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:53.191642    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:53.191853 ip-10-0-128-43 kubenswrapper[2582]: E0728 19:59:53.191843    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls podName:ec0121bc-2354-4424-9fd2-965012dcfc38 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:01.191802984 +0000 UTC m=+51.046114033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzjb6" (UID: "ec0121bc-2354-4424-9fd2-965012dcfc38") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:59:53.220930 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:53.220897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:53.304394 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:53.304308    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:53.405163 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:53.405128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:53.954819 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:53.954784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:54.246329 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:54.246275    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:54.308646 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:54.308610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:54.758418 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:54.758389    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:55.234532 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:55.234504    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:55.368024 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:55.367991    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:55.555880 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:55.555809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:55.755848 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:55.755817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:56.355067 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:56.355037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:56.480483 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:56.480453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:56.753012 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:56.752947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:56.955393 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:56.955364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:57.151719 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:57.151615    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:57.233013 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:57.232983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:57.552500 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:57.552463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:57.974209 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:57.974135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:58.754436 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:58.754404    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:58.875024 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:58.874994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:58.976476 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:58.976448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:59.360992 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:59.360957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 19:59:59.553734 ip-10-0-128-43 kubenswrapper[2582]: I0728 19:59:59.553692    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:00.168502 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.168457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:00.345072 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.345042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 20:00:00.345258 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.345127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 20:00:00.348382 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.348353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5339e904-5d69-4e23-bbc6-32bbda9e309d-metrics-tls\") pod \"dns-default-sp2jg\" (UID: \"5339e904-5d69-4e23-bbc6-32bbda9e309d\") " pod="openshift-dns/dns-default-sp2jg"
Jul 28 20:00:00.348501 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.348481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") pod \"image-registry-84844bb5c5-6bw2q\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") " pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 20:00:00.383151 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.383119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:00.433576 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.433506    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 20:00:00.442593 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.442562    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sp2jg"
Jul 28 20:00:00.584840 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.584811    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84844bb5c5-6bw2q"]
Jul 28 20:00:00.588196 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:00:00.588169    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53d5b18_7246_472c_a974_2d2d0e161f80.slice/crio-dbe3016fabc1673933a3ba19981ab31db618237ba7008abcfbfc007d6d6eb19e WatchSource:0}: Error finding container dbe3016fabc1673933a3ba19981ab31db618237ba7008abcfbfc007d6d6eb19e: Status 404 returned error can't find the container with id dbe3016fabc1673933a3ba19981ab31db618237ba7008abcfbfc007d6d6eb19e
Jul 28 20:00:00.604533 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.604506    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sp2jg"]
Jul 28 20:00:00.607203 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:00:00.607174    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5339e904_5d69_4e23_bbc6_32bbda9e309d.slice/crio-f297ed06112e4c0d26c21d9cae3a31a24547e24299b5eda78a792809f6b0ebf0 WatchSource:0}: Error finding container f297ed06112e4c0d26c21d9cae3a31a24547e24299b5eda78a792809f6b0ebf0: Status 404 returned error can't find the container with id f297ed06112e4c0d26c21d9cae3a31a24547e24299b5eda78a792809f6b0ebf0
Jul 28 20:00:00.762126 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.762100    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:00.838798 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.838764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sp2jg" event={"ID":"5339e904-5d69-4e23-bbc6-32bbda9e309d","Type":"ContainerStarted","Data":"f297ed06112e4c0d26c21d9cae3a31a24547e24299b5eda78a792809f6b0ebf0"}
Jul 28 20:00:00.840188 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.840159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q" event={"ID":"c53d5b18-7246-472c-a974-2d2d0e161f80","Type":"ContainerStarted","Data":"607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d"}
Jul 28 20:00:00.840317 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.840202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q" event={"ID":"c53d5b18-7246-472c-a974-2d2d0e161f80","Type":"ContainerStarted","Data":"dbe3016fabc1673933a3ba19981ab31db618237ba7008abcfbfc007d6d6eb19e"}
Jul 28 20:00:00.840359 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.840318    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 20:00:00.861488 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:00.861447    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q" podStartSLOduration=24.861432551 podStartE2EDuration="24.861432551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:00.860507119 +0000 UTC m=+50.714818185" watchObservedRunningTime="2026-07-28 20:00:00.861432551 +0000 UTC m=+50.715743617"
Jul 28 20:00:01.015138 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:01.015072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:01.173071 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:01.173033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:01.252865 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:01.252823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 20:00:01.255199 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:01.255172    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec0121bc-2354-4424-9fd2-965012dcfc38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzjb6\" (UID: \"ec0121bc-2354-4424-9fd2-965012dcfc38\") " pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 20:00:01.259169 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:01.259149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gzjb6"
Jul 28 20:00:01.422163 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:01.422131    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gzjb6"]
Jul 28 20:00:01.425362 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:00:01.425333    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0121bc_2354_4424_9fd2_965012dcfc38.slice/crio-2322cc6c5f4304a2a4cfce2a10c1e2d411685b73a28839f225b4e3d358834a5a WatchSource:0}: Error finding container 2322cc6c5f4304a2a4cfce2a10c1e2d411685b73a28839f225b4e3d358834a5a: Status 404 returned error can't find the container with id 2322cc6c5f4304a2a4cfce2a10c1e2d411685b73a28839f225b4e3d358834a5a
Jul 28 20:00:01.583550 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:01.583467    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:01.844760 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:01.844663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gzjb6" event={"ID":"ec0121bc-2354-4424-9fd2-965012dcfc38","Type":"ContainerStarted","Data":"2322cc6c5f4304a2a4cfce2a10c1e2d411685b73a28839f225b4e3d358834a5a"}
Jul 28 20:00:02.156141 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:02.156052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:02.557196 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:02.557159    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:02.959829 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:02.959801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:03.285504 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:03.285434    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:03.624582 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:03.624500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:03.853171 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:03.853136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sp2jg" event={"ID":"5339e904-5d69-4e23-bbc6-32bbda9e309d","Type":"ContainerStarted","Data":"9151cc8f9fcfff29f29207e82ff6792601e0e10ffb65aa232dd06fddf508b09f"}
Jul 28 20:00:03.853439 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:03.853197    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sp2jg" event={"ID":"5339e904-5d69-4e23-bbc6-32bbda9e309d","Type":"ContainerStarted","Data":"e3d6244222f03119fbfda616c71ec82a5a7d6d38581592377aa3df7fa4af58f1"}
Jul 28 20:00:03.853439 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:03.853306    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sp2jg"
Jul 28 20:00:03.871575 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:03.871519    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sp2jg" podStartSLOduration=17.682990083 podStartE2EDuration="19.87149869s" totalImagesPullingTime="2.188508607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:03.871496156 +0000 UTC m=+53.725807222" watchObservedRunningTime="2026-07-28 20:00:03.87149869 +0000 UTC m=+53.725809757"
Jul 28 20:00:07.544368 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.544336    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"]
Jul 28 20:00:07.547430 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.547407    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"]
Jul 28 20:00:07.547600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.547543    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"
Jul 28 20:00:07.550268 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.550244    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 20:00:07.550450 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.550429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"
Jul 28 20:00:07.550688 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.550639    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 20:00:07.551340 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.551314    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gghrt\""
Jul 28 20:00:07.551874 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.551856    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"]
Jul 28 20:00:07.555426 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.555410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.555513 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.555427    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 20:00:07.555855 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.555829    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 20:00:07.556518 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.556040    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 20:00:07.556701 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.556565    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-b5rvv\""
Jul 28 20:00:07.557067 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.557051    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 20:00:07.559674 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.559621    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"]
Jul 28 20:00:07.559772 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.559739    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 20:00:07.563142 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.563100    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"]
Jul 28 20:00:07.568419 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.568400    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"]
Jul 28 20:00:07.572751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.572402    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"]
Jul 28 20:00:07.572751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.572441    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84844bb5c5-6bw2q"]
Jul 28 20:00:07.572751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.572499    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.579343 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.579065    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 20:00:07.579343 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.579105    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 20:00:07.579992 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.579973    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 20:00:07.580343 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.580321    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 20:00:07.593855 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.593836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"]
Jul 28 20:00:07.603744 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.603723    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/557703ca-049a-45bb-805f-b1d1bd1b4d12-tmp\") pod \"klusterlet-addon-workmgr-79c8667c6-hfxgg\" (UID: \"557703ca-049a-45bb-805f-b1d1bd1b4d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.603851 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.603812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-hub\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.603851 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.603843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n8gh\" (UniqueName: \"kubernetes.io/projected/42ffd97b-0c13-48a8-8b4f-452256cfea7c-kube-api-access-2n8gh\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.603973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.603867    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb1a1641-bacf-4667-996b-71696f3c7afb-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6tlg5\" (UID: \"eb1a1641-bacf-4667-996b-71696f3c7afb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"
Jul 28 20:00:07.603973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.603889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/557703ca-049a-45bb-805f-b1d1bd1b4d12-klusterlet-config\") pod \"klusterlet-addon-workmgr-79c8667c6-hfxgg\" (UID: \"557703ca-049a-45bb-805f-b1d1bd1b4d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.603973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.603905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-ca\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.603973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.603932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbm59\" (UniqueName: \"kubernetes.io/projected/f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6-kube-api-access-wbm59\") pod \"managed-serviceaccount-addon-agent-6f69b46486-ds6n2\" (UID: \"f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"
Jul 28 20:00:07.604193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.604001    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.604193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.604020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f69b46486-ds6n2\" (UID: \"f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"
Jul 28 20:00:07.604193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.604043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb1a1641-bacf-4667-996b-71696f3c7afb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6tlg5\" (UID: \"eb1a1641-bacf-4667-996b-71696f3c7afb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"
Jul 28 20:00:07.604193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.604068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt579\" (UniqueName: \"kubernetes.io/projected/557703ca-049a-45bb-805f-b1d1bd1b4d12-kube-api-access-dt579\") pod \"klusterlet-addon-workmgr-79c8667c6-hfxgg\" (UID: \"557703ca-049a-45bb-805f-b1d1bd1b4d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.604193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.604091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.604193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.604117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ffd97b-0c13-48a8-8b4f-452256cfea7c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.671144 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.671118    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-97cb54f69-xgx74"]
Jul 28 20:00:07.674117 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.674089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.686629 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.686606    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:07.692103 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.692025    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97cb54f69-xgx74"]
Jul 28 20:00:07.705487 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b6d19d-531d-4167-99d9-a45d465186f1-bound-sa-token\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.705613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt579\" (UniqueName: \"kubernetes.io/projected/557703ca-049a-45bb-805f-b1d1bd1b4d12-kube-api-access-dt579\") pod \"klusterlet-addon-workmgr-79c8667c6-hfxgg\" (UID: \"557703ca-049a-45bb-805f-b1d1bd1b4d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.705613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.705613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ffd97b-0c13-48a8-8b4f-452256cfea7c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.705613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b6d19d-531d-4167-99d9-a45d465186f1-image-registry-private-configuration\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.705819 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b6d19d-531d-4167-99d9-a45d465186f1-ca-trust-extracted\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.705819 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/557703ca-049a-45bb-805f-b1d1bd1b4d12-tmp\") pod \"klusterlet-addon-workmgr-79c8667c6-hfxgg\" (UID: \"557703ca-049a-45bb-805f-b1d1bd1b4d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.705819 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkjr\" (UniqueName: \"kubernetes.io/projected/28b6d19d-531d-4167-99d9-a45d465186f1-kube-api-access-nwkjr\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.705819 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b6d19d-531d-4167-99d9-a45d465186f1-installation-pull-secrets\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.705819 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-hub\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.705819 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n8gh\" (UniqueName: \"kubernetes.io/projected/42ffd97b-0c13-48a8-8b4f-452256cfea7c-kube-api-access-2n8gh\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.706102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb1a1641-bacf-4667-996b-71696f3c7afb-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6tlg5\" (UID: \"eb1a1641-bacf-4667-996b-71696f3c7afb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"
Jul 28 20:00:07.706102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/557703ca-049a-45bb-805f-b1d1bd1b4d12-klusterlet-config\") pod \"klusterlet-addon-workmgr-79c8667c6-hfxgg\" (UID: \"557703ca-049a-45bb-805f-b1d1bd1b4d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.706102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b6d19d-531d-4167-99d9-a45d465186f1-registry-certificates\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.706102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-ca\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.706102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b6d19d-531d-4167-99d9-a45d465186f1-trusted-ca\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.706102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.705992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbm59\" (UniqueName: \"kubernetes.io/projected/f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6-kube-api-access-wbm59\") pod \"managed-serviceaccount-addon-agent-6f69b46486-ds6n2\" (UID: \"f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"
Jul 28 20:00:07.706102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.706046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.706102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.706075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f69b46486-ds6n2\" (UID: \"f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"
Jul 28 20:00:07.706102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.706100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb1a1641-bacf-4667-996b-71696f3c7afb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6tlg5\" (UID: \"eb1a1641-bacf-4667-996b-71696f3c7afb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"
Jul 28 20:00:07.706544 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.706131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b6d19d-531d-4167-99d9-a45d465186f1-registry-tls\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.707326 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.707284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/557703ca-049a-45bb-805f-b1d1bd1b4d12-tmp\") pod \"klusterlet-addon-workmgr-79c8667c6-hfxgg\" (UID: \"557703ca-049a-45bb-805f-b1d1bd1b4d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.707485 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.707462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb1a1641-bacf-4667-996b-71696f3c7afb-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6tlg5\" (UID: \"eb1a1641-bacf-4667-996b-71696f3c7afb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"
Jul 28 20:00:07.707961 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.707921    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ffd97b-0c13-48a8-8b4f-452256cfea7c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.709988 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.709948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.710758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.710717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.712039 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.712014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb1a1641-bacf-4667-996b-71696f3c7afb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6tlg5\" (UID: \"eb1a1641-bacf-4667-996b-71696f3c7afb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"
Jul 28 20:00:07.712587 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.712565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-hub\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.713036 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.713013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/42ffd97b-0c13-48a8-8b4f-452256cfea7c-ca\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.714685 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.714659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f69b46486-ds6n2\" (UID: \"f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"
Jul 28 20:00:07.714863 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.714844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/557703ca-049a-45bb-805f-b1d1bd1b4d12-klusterlet-config\") pod \"klusterlet-addon-workmgr-79c8667c6-hfxgg\" (UID: \"557703ca-049a-45bb-805f-b1d1bd1b4d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.723897 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.723867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt579\" (UniqueName: \"kubernetes.io/projected/557703ca-049a-45bb-805f-b1d1bd1b4d12-kube-api-access-dt579\") pod \"klusterlet-addon-workmgr-79c8667c6-hfxgg\" (UID: \"557703ca-049a-45bb-805f-b1d1bd1b4d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.724979 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.724919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbm59\" (UniqueName: \"kubernetes.io/projected/f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6-kube-api-access-wbm59\") pod \"managed-serviceaccount-addon-agent-6f69b46486-ds6n2\" (UID: \"f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"
Jul 28 20:00:07.725750 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.725698    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n8gh\" (UniqueName: \"kubernetes.io/projected/42ffd97b-0c13-48a8-8b4f-452256cfea7c-kube-api-access-2n8gh\") pod \"cluster-proxy-proxy-agent-659548875-xrfsg\" (UID: \"42ffd97b-0c13-48a8-8b4f-452256cfea7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:07.805173 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.805055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:07.806742 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.806712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b6d19d-531d-4167-99d9-a45d465186f1-registry-tls\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.806877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.806754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b6d19d-531d-4167-99d9-a45d465186f1-bound-sa-token\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.806877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.806785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b6d19d-531d-4167-99d9-a45d465186f1-image-registry-private-configuration\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.806877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.806814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b6d19d-531d-4167-99d9-a45d465186f1-ca-trust-extracted\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.806877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.806866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkjr\" (UniqueName: \"kubernetes.io/projected/28b6d19d-531d-4167-99d9-a45d465186f1-kube-api-access-nwkjr\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.807093 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.806906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b6d19d-531d-4167-99d9-a45d465186f1-installation-pull-secrets\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.807093 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.806955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b6d19d-531d-4167-99d9-a45d465186f1-registry-certificates\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.807093 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.806984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b6d19d-531d-4167-99d9-a45d465186f1-trusted-ca\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.809892 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.808510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b6d19d-531d-4167-99d9-a45d465186f1-trusted-ca\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.809892 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.808872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b6d19d-531d-4167-99d9-a45d465186f1-ca-trust-extracted\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.809892 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.809848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b6d19d-531d-4167-99d9-a45d465186f1-registry-certificates\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.812179 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.812153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b6d19d-531d-4167-99d9-a45d465186f1-registry-tls\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.812834 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.812794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b6d19d-531d-4167-99d9-a45d465186f1-image-registry-private-configuration\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.813148 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.813125    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b6d19d-531d-4167-99d9-a45d465186f1-installation-pull-secrets\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.822396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.822356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b6d19d-531d-4167-99d9-a45d465186f1-bound-sa-token\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.825924 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.825889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkjr\" (UniqueName: \"kubernetes.io/projected/28b6d19d-531d-4167-99d9-a45d465186f1-kube-api-access-nwkjr\") pod \"image-registry-97cb54f69-xgx74\" (UID: \"28b6d19d-531d-4167-99d9-a45d465186f1\") " pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:07.861042 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.860785    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"
Jul 28 20:00:07.871506 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.871472    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gzjb6" event={"ID":"ec0121bc-2354-4424-9fd2-965012dcfc38","Type":"ContainerStarted","Data":"dd3008e7bc930a40f012edfe66550ac6e279640d4578e49f0202c57615fcb8e0"}
Jul 28 20:00:07.877952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.877931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"
Jul 28 20:00:07.887310 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.887267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:07.895105 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:07.894928    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"
Jul 28 20:00:08.003933 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.003645    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:08.064157 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.058478    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5"]
Jul 28 20:00:08.078210 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.078097    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"]
Jul 28 20:00:08.097016 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.096907    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2"]
Jul 28 20:00:08.122454 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.121476    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg"]
Jul 28 20:00:08.125873 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:00:08.125764    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42ffd97b_0c13_48a8_8b4f_452256cfea7c.slice/crio-19d22bdb7038f1141baf509bbc0155bd71f9285261aa743f121bfa08b067115c WatchSource:0}: Error finding container 19d22bdb7038f1141baf509bbc0155bd71f9285261aa743f121bfa08b067115c: Status 404 returned error can't find the container with id 19d22bdb7038f1141baf509bbc0155bd71f9285261aa743f121bfa08b067115c
Jul 28 20:00:08.178511 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.178428    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97cb54f69-xgx74"]
Jul 28 20:00:08.318953 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.318881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:08.376676 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.376648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:08.878493 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.878327    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cb54f69-xgx74" event={"ID":"28b6d19d-531d-4167-99d9-a45d465186f1","Type":"ContainerStarted","Data":"ae6125570ebb5c1306ae01c40c442ea46c1b0f454ca78e8de8d5f97a22e4e048"}
Jul 28 20:00:08.878493 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.878394    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:08.878493 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.878416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cb54f69-xgx74" event={"ID":"28b6d19d-531d-4167-99d9-a45d465186f1","Type":"ContainerStarted","Data":"7a4fc8ca2eb1b023845eb0f47383b712265099e8dda967543f28c1615482a531"}
Jul 28 20:00:08.880186 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.880144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2" event={"ID":"f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6","Type":"ContainerStarted","Data":"b0ac8ec3685644da0071f9eb3caa2cb0b36f9c3002f343a08de8967256bc86a9"}
Jul 28 20:00:08.881342 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.881319    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5" event={"ID":"eb1a1641-bacf-4667-996b-71696f3c7afb","Type":"ContainerStarted","Data":"5a8281f8825e3b81cc4293958e36c6b5a9cfeb8f8d3ea14cfd1294baa69f0bfa"}
Jul 28 20:00:08.882640 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.882610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg" event={"ID":"42ffd97b-0c13-48a8-8b4f-452256cfea7c","Type":"ContainerStarted","Data":"19d22bdb7038f1141baf509bbc0155bd71f9285261aa743f121bfa08b067115c"}
Jul 28 20:00:08.884403 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.884381    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg" event={"ID":"557703ca-049a-45bb-805f-b1d1bd1b4d12","Type":"ContainerStarted","Data":"5f9e74e8732fad703a223a89da65d276875a8507e75b09d596a289f5539651a0"}
Jul 28 20:00:08.902765 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:08.902697    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-97cb54f69-xgx74" podStartSLOduration=1.902678895 podStartE2EDuration="1.902678895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:08.900502425 +0000 UTC m=+58.754813515" watchObservedRunningTime="2026-07-28 20:00:08.902678895 +0000 UTC m=+58.756989961"
Jul 28 20:00:09.267546 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:09.267508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:09.318416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:09.318385    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:10.422052 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:10.422025    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:10.481064 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:10.481024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:10.545772 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:10.545710    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:10.695284 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:10.695203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:10.800046 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:10.799935    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9lg5"
Jul 28 20:00:10.995709 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:10.995635    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:11.269456 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:11.269367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:11.598975 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:11.598801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:11.700114 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:11.700079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:11.882007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:11.881935    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:12.025969 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:12.025934    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:12.123472 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:12.123437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:12.823022 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:12.822933    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:12.947215 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:12.947150    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:13.859076 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:13.859047    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sp2jg"
Jul 28 20:00:15.319925 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:15.319879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:15.421761 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:15.421709    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:15.490833 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:15.490806    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:15.670175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:15.670087    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:16.283711 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.283675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 20:00:16.286281 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.286262    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 20:00:16.296214 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.296195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dedd29f8-8425-42cc-804d-2b786679bd20-original-pull-secret\") pod \"global-pull-secret-syncer-pcwfl\" (UID: \"dedd29f8-8425-42cc-804d-2b786679bd20\") " pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 20:00:16.384659 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.384629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 20:00:16.387312 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.387272    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 20:00:16.397803 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.397782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f23cdda-4df1-4276-8197-80943b7648b0-metrics-certs\") pod \"network-metrics-daemon-nnc2m\" (UID: \"6f23cdda-4df1-4276-8197-80943b7648b0\") " pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 20:00:16.402720 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.402697    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcwfl"
Jul 28 20:00:16.485902 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.485866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 20:00:16.492113 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.492073    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 20:00:16.498458 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.498431    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 20:00:16.508951 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.508927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4wn\" (UniqueName: \"kubernetes.io/projected/b8f79db1-5f9e-4832-8647-08a142ef88e0-kube-api-access-jf4wn\") pod \"network-check-target-fh5cb\" (UID: \"b8f79db1-5f9e-4832-8647-08a142ef88e0\") " pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 20:00:16.525687 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.525664    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcwfl"]
Jul 28 20:00:16.528562 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:00:16.528538    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedd29f8_8425_42cc_804d_2b786679bd20.slice/crio-694e07b47d3a72744fbef13746345cf38ac65a5911d854b0cf7788cd48b99987 WatchSource:0}: Error finding container 694e07b47d3a72744fbef13746345cf38ac65a5911d854b0cf7788cd48b99987: Status 404 returned error can't find the container with id 694e07b47d3a72744fbef13746345cf38ac65a5911d854b0cf7788cd48b99987
Jul 28 20:00:16.692258 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.692226    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kg9b\""
Jul 28 20:00:16.698902 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.698875    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gjwl7\""
Jul 28 20:00:16.700171 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.700150    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 20:00:16.707593 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.707573    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nnc2m"
Jul 28 20:00:16.852172 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.852139    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nnc2m"]
Jul 28 20:00:16.855959 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:00:16.855934    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f23cdda_4df1_4276_8197_80943b7648b0.slice/crio-53d172a13fdde444305faf2873dcfaf3da06075b5dc7f3fb390121f5329d2a00 WatchSource:0}: Error finding container 53d172a13fdde444305faf2873dcfaf3da06075b5dc7f3fb390121f5329d2a00: Status 404 returned error can't find the container with id 53d172a13fdde444305faf2873dcfaf3da06075b5dc7f3fb390121f5329d2a00
Jul 28 20:00:16.871742 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.871720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fh5cb"]
Jul 28 20:00:16.875378 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:00:16.875348    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f79db1_5f9e_4832_8647_08a142ef88e0.slice/crio-9d7ec59a4725b7319e9677c647a8518d69ee317ed326e737c8bd86bcb453ab56 WatchSource:0}: Error finding container 9d7ec59a4725b7319e9677c647a8518d69ee317ed326e737c8bd86bcb453ab56: Status 404 returned error can't find the container with id 9d7ec59a4725b7319e9677c647a8518d69ee317ed326e737c8bd86bcb453ab56
Jul 28 20:00:16.924322 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.923902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5" event={"ID":"eb1a1641-bacf-4667-996b-71696f3c7afb","Type":"ContainerStarted","Data":"d4016161187c903d1baf9b650cad8dd374d5ef373afcb12ccaf9b5f0e9dd8d33"}
Jul 28 20:00:16.925800 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.925771    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nnc2m" event={"ID":"6f23cdda-4df1-4276-8197-80943b7648b0","Type":"ContainerStarted","Data":"53d172a13fdde444305faf2873dcfaf3da06075b5dc7f3fb390121f5329d2a00"}
Jul 28 20:00:16.927997 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.927975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg" event={"ID":"42ffd97b-0c13-48a8-8b4f-452256cfea7c","Type":"ContainerStarted","Data":"657666ab5199cf455d602859096b989b9b8eed1d077571a9d3b87055fee75035"}
Jul 28 20:00:16.928126 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.928010    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg" event={"ID":"42ffd97b-0c13-48a8-8b4f-452256cfea7c","Type":"ContainerStarted","Data":"134f707485f14309cba8a9a852e66a775c62d0d795c4078831f7778a18b9924b"}
Jul 28 20:00:16.928126 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.928029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg" event={"ID":"42ffd97b-0c13-48a8-8b4f-452256cfea7c","Type":"ContainerStarted","Data":"2b0e2e7dd1f0245491be3568133d8783ff5c537bb4a7675f637bf97e4ea3346e"}
Jul 28 20:00:16.929643 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.929618    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg" event={"ID":"557703ca-049a-45bb-805f-b1d1bd1b4d12","Type":"ContainerStarted","Data":"dda656fb2edc3e6d181e1da394b360cf31d73ecc4fe9af6f69e8c42a51efd6e1"}
Jul 28 20:00:16.929862 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.929843    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:16.930939 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.930915    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fh5cb" event={"ID":"b8f79db1-5f9e-4832-8647-08a142ef88e0","Type":"ContainerStarted","Data":"9d7ec59a4725b7319e9677c647a8518d69ee317ed326e737c8bd86bcb453ab56"}
Jul 28 20:00:16.931620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.931603    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg"
Jul 28 20:00:16.932959 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.932936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gzjb6" event={"ID":"ec0121bc-2354-4424-9fd2-965012dcfc38","Type":"ContainerStarted","Data":"5711d8d603dd919f60aa77a40c730ff5604f713f83669e0ec3507cf9570a0214"}
Jul 28 20:00:16.934347 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.934324    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2" event={"ID":"f7aa1bed-e53d-4ee1-8588-9e3fea4b45e6","Type":"ContainerStarted","Data":"47237ebf645ae41fc27e3ee82edeb069f603dca7a6eb588153a44ad29567ec5b"}
Jul 28 20:00:16.935485 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.935461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pcwfl" event={"ID":"dedd29f8-8425-42cc-804d-2b786679bd20","Type":"ContainerStarted","Data":"694e07b47d3a72744fbef13746345cf38ac65a5911d854b0cf7788cd48b99987"}
Jul 28 20:00:16.984015 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:16.983872    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6tlg5" podStartSLOduration=2.068296157 podStartE2EDuration="9.983855427s" totalImagesPullingTime="7.91555927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:16.983803098 +0000 UTC m=+66.838114164" watchObservedRunningTime="2026-07-28 20:00:16.983855427 +0000 UTC m=+66.838166494"
Jul 28 20:00:17.002754 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:17.002105    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659548875-xrfsg" podStartSLOduration=2.150058579 podStartE2EDuration="10.002086912s" totalImagesPullingTime="7.852028333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:17.002006089 +0000 UTC m=+66.856317156" watchObservedRunningTime="2026-07-28 20:00:17.002086912 +0000 UTC m=+66.856397978"
Jul 28 20:00:17.019540 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:17.019457    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c8667c6-hfxgg" podStartSLOduration=2.111589318 podStartE2EDuration="10.019439518s" totalImagesPullingTime="7.9078502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:17.018355834 +0000 UTC m=+66.872666901" watchObservedRunningTime="2026-07-28 20:00:17.019439518 +0000 UTC m=+66.873750581"
Jul 28 20:00:17.035195 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:17.035146    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f69b46486-ds6n2" podStartSLOduration=2.149653107 podStartE2EDuration="10.035130614s" totalImagesPullingTime="7.885477507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:17.03395301 +0000 UTC m=+66.888264074" watchObservedRunningTime="2026-07-28 20:00:17.035130614 +0000 UTC m=+66.889441661"
Jul 28 20:00:17.090860 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:17.090828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:17.152840 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:17.152809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:17.579577 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:17.579285    2582 patch_prober.go:28] interesting pod/image-registry-84844bb5c5-6bw2q container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 20:00:17.579577 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:17.579381    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q" podUID="c53d5b18-7246-472c-a974-2d2d0e161f80" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:00:18.368584 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:18.368546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:18.440041 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:18.439839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:22.810428 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:22.810376    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:23.389878 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:23.389798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:23.490614 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:23.490583    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:23.972828 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:23.972793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gzjb6" event={"ID":"ec0121bc-2354-4424-9fd2-965012dcfc38","Type":"ContainerStarted","Data":"fa10dbbf32418bc607298af2385dc6f8cc16806ce8582eece4b6505363c2dc77"}
Jul 28 20:00:23.974265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:23.974239    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pcwfl" event={"ID":"dedd29f8-8425-42cc-804d-2b786679bd20","Type":"ContainerStarted","Data":"fca5842ae0f1298e58901928ea5ecfa2660f255363a428636921e6066c894497"}
Jul 28 20:00:23.975818 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:23.975782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nnc2m" event={"ID":"6f23cdda-4df1-4276-8197-80943b7648b0","Type":"ContainerStarted","Data":"56692bb3402fa0876ff02fd93b2e3e380fd48821660d8141604cc48fa0a44fd3"}
Jul 28 20:00:23.975931 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:23.975902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nnc2m" event={"ID":"6f23cdda-4df1-4276-8197-80943b7648b0","Type":"ContainerStarted","Data":"81e8ad2e900e779b579930bb04a7b0d9a2941a05cf8ef76e19fb27c0ef0439e6"}
Jul 28 20:00:23.977136 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:23.977116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fh5cb" event={"ID":"b8f79db1-5f9e-4832-8647-08a142ef88e0","Type":"ContainerStarted","Data":"7355e65666bb127d58c656ec11a5ab2ab125833fee6d8f2983711553caada655"}
Jul 28 20:00:23.977286 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:23.977271    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 20:00:23.997832 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:23.997785    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gzjb6" podStartSLOduration=18.920375232 podStartE2EDuration="38.997772817s" totalImagesPullingTime="20.077397585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:23.996440519 +0000 UTC m=+73.850751585" watchObservedRunningTime="2026-07-28 20:00:23.997772817 +0000 UTC m=+73.852083883"
Jul 28 20:00:24.027829 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:24.027784    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nnc2m" podStartSLOduration=67.989217399 podStartE2EDuration="1m14.027768331s" totalImagesPullingTime="6.038550932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:24.027650834 +0000 UTC m=+73.881961903" watchObservedRunningTime="2026-07-28 20:00:24.027768331 +0000 UTC m=+73.882079399"
Jul 28 20:00:24.028488 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:24.028457    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pcwfl" podStartSLOduration=67.6491765 podStartE2EDuration="1m14.028449113s" totalImagesPullingTime="6.379272613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:24.012135028 +0000 UTC m=+73.866446148" watchObservedRunningTime="2026-07-28 20:00:24.028449113 +0000 UTC m=+73.882760180"
Jul 28 20:00:24.188342 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:24.188314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:24.286829 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:24.286804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:25.489118 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:25.489081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:25.546972 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:25.546943    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:26.483725 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:26.483693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:27.408858 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:27.408831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:27.463733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:27.463703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:27.577404 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:27.577379    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 20:00:27.598200 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:27.598152    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fh5cb" podStartSLOduration=71.570921755 podStartE2EDuration="1m17.598136967s" totalImagesPullingTime="6.027215212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:24.047008375 +0000 UTC m=+73.901319443" watchObservedRunningTime="2026-07-28 20:00:27.598136967 +0000 UTC m=+77.452448052"
Jul 28 20:00:29.894120 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:29.894092    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-97cb54f69-xgx74"
Jul 28 20:00:30.037467 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:30.037431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:30.140546 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:30.140514    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:30.237500 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:30.237476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:30.512067 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:30.511980    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:31.464245 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:31.464202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:32.328021 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.327953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:32.601465 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.601327    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q" podUID="c53d5b18-7246-472c-a974-2d2d0e161f80" containerName="registry" containerID="cri-o://607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d" gracePeriod=30
Jul 28 20:00:32.747624 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.747594    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:32.835673 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.835651    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 20:00:32.847508 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.847482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:32.922188 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-certificates\") pod \"c53d5b18-7246-472c-a974-2d2d0e161f80\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") "
Jul 28 20:00:32.922188 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922157    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-bound-sa-token\") pod \"c53d5b18-7246-472c-a974-2d2d0e161f80\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") "
Jul 28 20:00:32.922188 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922178    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-trusted-ca\") pod \"c53d5b18-7246-472c-a974-2d2d0e161f80\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") "
Jul 28 20:00:32.922448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922229    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") pod \"c53d5b18-7246-472c-a974-2d2d0e161f80\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") "
Jul 28 20:00:32.922448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922356    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7kzx\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-kube-api-access-p7kzx\") pod \"c53d5b18-7246-472c-a974-2d2d0e161f80\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") "
Jul 28 20:00:32.922448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922400    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c53d5b18-7246-472c-a974-2d2d0e161f80-ca-trust-extracted\") pod \"c53d5b18-7246-472c-a974-2d2d0e161f80\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") "
Jul 28 20:00:32.922448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922437    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-installation-pull-secrets\") pod \"c53d5b18-7246-472c-a974-2d2d0e161f80\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") "
Jul 28 20:00:32.922716 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922487    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-image-registry-private-configuration\") pod \"c53d5b18-7246-472c-a974-2d2d0e161f80\" (UID: \"c53d5b18-7246-472c-a974-2d2d0e161f80\") "
Jul 28 20:00:32.922716 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922538    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c53d5b18-7246-472c-a974-2d2d0e161f80" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:00:32.922716 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922668    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c53d5b18-7246-472c-a974-2d2d0e161f80" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:00:32.922868 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922725    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-certificates\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:00:32.922868 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.922743    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c53d5b18-7246-472c-a974-2d2d0e161f80-trusted-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:00:32.929947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.929899    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c53d5b18-7246-472c-a974-2d2d0e161f80" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:00:32.930047 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.929937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c53d5b18-7246-472c-a974-2d2d0e161f80" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:00:32.930047 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.929985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c53d5b18-7246-472c-a974-2d2d0e161f80" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:00:32.934092 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.934062    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53d5b18-7246-472c-a974-2d2d0e161f80-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c53d5b18-7246-472c-a974-2d2d0e161f80" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:00:32.936094 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.936068    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-kube-api-access-p7kzx" (OuterVolumeSpecName: "kube-api-access-p7kzx") pod "c53d5b18-7246-472c-a974-2d2d0e161f80" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80"). InnerVolumeSpecName "kube-api-access-p7kzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:00:32.936186 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.936131    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c53d5b18-7246-472c-a974-2d2d0e161f80" (UID: "c53d5b18-7246-472c-a974-2d2d0e161f80"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:00:32.941774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.941746    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6lk77"]
Jul 28 20:00:32.942046 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.942031    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c53d5b18-7246-472c-a974-2d2d0e161f80" containerName="registry"
Jul 28 20:00:32.942125 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.942049    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c53d5b18-7246-472c-a974-2d2d0e161f80" containerName="registry"
Jul 28 20:00:32.942125 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.942111    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c53d5b18-7246-472c-a974-2d2d0e161f80" containerName="registry"
Jul 28 20:00:32.946610 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.946592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:32.949213 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.949067    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 20:00:32.949213 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.949090    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 20:00:32.949213 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.949120    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q6dkm\""
Jul 28 20:00:32.949213 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.949068    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 20:00:32.949868 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.949429    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 20:00:32.949868 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.949492    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 20:00:32.949868 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:32.949617    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 20:00:33.007727 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.007696    2582 generic.go:356] "Generic (PLEG): container finished" podID="c53d5b18-7246-472c-a974-2d2d0e161f80" containerID="607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d" exitCode=0
Jul 28 20:00:33.007871 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.007737    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q"
Jul 28 20:00:33.007871 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.007772    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q" event={"ID":"c53d5b18-7246-472c-a974-2d2d0e161f80","Type":"ContainerDied","Data":"607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d"}
Jul 28 20:00:33.007871 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.007813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84844bb5c5-6bw2q" event={"ID":"c53d5b18-7246-472c-a974-2d2d0e161f80","Type":"ContainerDied","Data":"dbe3016fabc1673933a3ba19981ab31db618237ba7008abcfbfc007d6d6eb19e"}
Jul 28 20:00:33.007871 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.007834    2582 scope.go:117] "RemoveContainer" containerID="607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d"
Jul 28 20:00:33.016265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.016247    2582 scope.go:117] "RemoveContainer" containerID="607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d"
Jul 28 20:00:33.016592 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:00:33.016574    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d\": container with ID starting with 607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d not found: ID does not exist" containerID="607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d"
Jul 28 20:00:33.016653 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.016601    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d"} err="failed to get container status \"607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d\": rpc error: code = NotFound desc = could not find container \"607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d\": container with ID starting with 607e9c60bae3fb151d5fbdfb82efac6da754ad4dbe5a923a29a412cdd2a4648d not found: ID does not exist"
Jul 28 20:00:33.023846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.023825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.023930 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.023858    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9084be9d-cb5f-43d9-bac6-187e91e3f73d-sys\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.023930 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.023875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-wtmp\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.024002 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.023930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8ml\" (UniqueName: \"kubernetes.io/projected/9084be9d-cb5f-43d9-bac6-187e91e3f73d-kube-api-access-5l8ml\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.024002 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.023973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-tls\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.024065 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-textfile\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.024065 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.024126 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9084be9d-cb5f-43d9-bac6-187e91e3f73d-root\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.024126 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9084be9d-cb5f-43d9-bac6-187e91e3f73d-metrics-client-ca\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.024196 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024137    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-image-registry-private-configuration\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:00:33.024196 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024154    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-bound-sa-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:00:33.024196 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024168    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-registry-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:00:33.024196 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024182    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7kzx\" (UniqueName: \"kubernetes.io/projected/c53d5b18-7246-472c-a974-2d2d0e161f80-kube-api-access-p7kzx\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:00:33.024350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024198    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c53d5b18-7246-472c-a974-2d2d0e161f80-ca-trust-extracted\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:00:33.024350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.024212    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c53d5b18-7246-472c-a974-2d2d0e161f80-installation-pull-secrets\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:00:33.033101 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.033078    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84844bb5c5-6bw2q"]
Jul 28 20:00:33.036819 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.036800    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84844bb5c5-6bw2q"]
Jul 28 20:00:33.125009 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.124982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125160 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9084be9d-cb5f-43d9-bac6-187e91e3f73d-sys\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125160 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-wtmp\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125160 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8ml\" (UniqueName: \"kubernetes.io/projected/9084be9d-cb5f-43d9-bac6-187e91e3f73d-kube-api-access-5l8ml\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125160 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-tls\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125160 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9084be9d-cb5f-43d9-bac6-187e91e3f73d-sys\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125160 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-textfile\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125181    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-wtmp\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9084be9d-cb5f-43d9-bac6-187e91e3f73d-root\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9084be9d-cb5f-43d9-bac6-187e91e3f73d-metrics-client-ca\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125654 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125493    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9084be9d-cb5f-43d9-bac6-187e91e3f73d-root\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125709 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-textfile\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125815 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.125979 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.125959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9084be9d-cb5f-43d9-bac6-187e91e3f73d-metrics-client-ca\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.127443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.127423    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.127733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.127715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9084be9d-cb5f-43d9-bac6-187e91e3f73d-node-exporter-tls\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.132947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.132927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8ml\" (UniqueName: \"kubernetes.io/projected/9084be9d-cb5f-43d9-bac6-187e91e3f73d-kube-api-access-5l8ml\") pod \"node-exporter-6lk77\" (UID: \"9084be9d-cb5f-43d9-bac6-187e91e3f73d\") " pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.201195 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.201138    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:33.257614 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.257593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6lk77"
Jul 28 20:00:33.261007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.260986    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:33.268816 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:00:33.268796    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9084be9d_cb5f_43d9_bac6_187e91e3f73d.slice/crio-4f1782fa2a3456d809b38cd7e7557855669fbf3eaac1ea9573648fb3f9f4efe2 WatchSource:0}: Error finding container 4f1782fa2a3456d809b38cd7e7557855669fbf3eaac1ea9573648fb3f9f4efe2: Status 404 returned error can't find the container with id 4f1782fa2a3456d809b38cd7e7557855669fbf3eaac1ea9573648fb3f9f4efe2
Jul 28 20:00:33.514637 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.514604    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:33.758800 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.758763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:33.875820 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.875740    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:33.983577 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:33.983543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:34.013596 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:34.013565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lk77" event={"ID":"9084be9d-cb5f-43d9-bac6-187e91e3f73d","Type":"ContainerStarted","Data":"4f1782fa2a3456d809b38cd7e7557855669fbf3eaac1ea9573648fb3f9f4efe2"}
Jul 28 20:00:34.582931 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:34.582907    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c53d5b18-7246-472c-a974-2d2d0e161f80" path="/var/lib/kubelet/pods/c53d5b18-7246-472c-a974-2d2d0e161f80/volumes"
Jul 28 20:00:34.778677 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:34.778592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:35.018890 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:35.018861    2582 generic.go:356] "Generic (PLEG): container finished" podID="9084be9d-cb5f-43d9-bac6-187e91e3f73d" containerID="ad811377da3bb2f583a33ed1f39fc9e576df729a6ae54b703edce1e7c3576562" exitCode=0
Jul 28 20:00:35.019050 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:35.018895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lk77" event={"ID":"9084be9d-cb5f-43d9-bac6-187e91e3f73d","Type":"ContainerDied","Data":"ad811377da3bb2f583a33ed1f39fc9e576df729a6ae54b703edce1e7c3576562"}
Jul 28 20:00:35.844076 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:35.844043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:35.945205 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:35.945173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:36.026954 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:36.026918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lk77" event={"ID":"9084be9d-cb5f-43d9-bac6-187e91e3f73d","Type":"ContainerStarted","Data":"2512baf9fd5b72e4cc4d4caac8090fe04998f007e54d7b3b48d8579f1382b796"}
Jul 28 20:00:36.027130 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:36.026982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lk77" event={"ID":"9084be9d-cb5f-43d9-bac6-187e91e3f73d","Type":"ContainerStarted","Data":"1ba3da1a771ee815c62685ae6af82f461b2c0a118b0752dd2991dbaff47ae8ee"}
Jul 28 20:00:36.042237 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:36.042199    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:36.050547 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:36.050500    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6lk77" podStartSLOduration=2.7496157869999998 podStartE2EDuration="4.050482178s" totalImagesPullingTime="1.238871972s" totalInitContainerRuntime="61.994419ms" isStatefulPod=true podCreationTimestamp="2026-07-28 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:36.049316223 +0000 UTC m=+85.903627282" watchObservedRunningTime="2026-07-28 20:00:36.050482178 +0000 UTC m=+85.904793247"
Jul 28 20:00:36.279869 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:36.279835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:37.665681 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:37.665650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:37.761590 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:37.761566    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:38.235871 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:38.235839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:39.093200 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:39.093167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:39.274346 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:39.274313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:39.374880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:39.374819    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:40.544370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:40.544337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:40.605645 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:40.605612    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:41.005801 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:41.005770    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:41.163043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:41.163007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:41.304696 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:41.304623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:42.508920 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:42.508890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:42.666928 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:42.666898    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:42.767189 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:42.767102    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:42.835460 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:42.835428    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:43.346587 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:43.346547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:43.447364 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:43.447334    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:49.804525 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:49.804464    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:49.998908 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:49.998880    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:50.275499 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:50.275449    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:54.256106 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:54.256073    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:54.345209 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:54.345180    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:54.983489 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:00:54.983459    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fh5cb"
Jul 28 20:01:10.189899 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:01:10.189866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:01:52.153434 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:01:52.153396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:01:52.640454 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:01:52.640390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:01:54.486519 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:01:54.486484    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:02:00.939442 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:02:00.939401    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:02:01.089814 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:02:01.089778    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:02:12.204652 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:02:12.204617    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:02:12.365454 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:02:12.365421    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:02:12.424992 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:02:12.424965    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:02:40.265521 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:02:40.265487    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:02:40.322025 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:02:40.322001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:07.336969 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:07.336929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:07.414973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:07.414939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:07.535589 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:07.535556    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:07.594529 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:07.594461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:18.426780 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:18.426745    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:18.483441 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:18.483411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:18.552175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:18.552145    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:19.229995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:19.229956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:19.283845 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:19.283811    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:25.472066 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:25.472031    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:25.528582 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:25.528551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:03:38.486231 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:03:38.486148    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:08.327090 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:08.327056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:10.211213 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:10.211174    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:10.455498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:10.455471    2582 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 20:04:34.993641 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:34.993604    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:35.891130 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:35.891095    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:38.843472 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:38.843426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:38.943082 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:38.943048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:39.845649 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:39.845618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:39.947200 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:39.947166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:40.112512 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:40.112426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:40.259259 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:40.259228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:42.222251 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:42.222216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:42.638078 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:42.638051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:45.603974 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:45.603940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:45.661942 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:45.661913    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:48.940210 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:48.940116    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:49.859130 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:49.859095    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:55.578005 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:04:55.577970    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:02.006449 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:02.006416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:02.108032 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:02.107999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:08.565847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:08.565809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:08.666034 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:08.665995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:10.460688 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:10.460658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:13.335401 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:13.335371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:13.435248 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:13.435220    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:22.421786 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:22.421751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:22.524784 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:22.524754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:22.700503 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:22.700427    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:22.807349 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:22.807319    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:23.102742 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.102711    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:23.400662 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.400575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:23.499114 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.499084    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"]
Jul 28 20:05:23.499763 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.499742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:23.502178 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.502162    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.504876 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.504853    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 20:05:23.504876 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.504868    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-npzt2\""
Jul 28 20:05:23.505223 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.505208    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 20:05:23.505377 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.505363    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 20:05:23.516979 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.516959    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 20:05:23.538864 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.538839    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"]
Jul 28 20:05:23.555843 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.555813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d323113d-b1bc-44d2-9eec-68bb234bd7ec-manifests\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.555958 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.555854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d323113d-b1bc-44d2-9eec-68bb234bd7ec-tmp\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.555958 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.555929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d323113d-b1bc-44d2-9eec-68bb234bd7ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.556079 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.555992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7nt\" (UniqueName: \"kubernetes.io/projected/d323113d-b1bc-44d2-9eec-68bb234bd7ec-kube-api-access-6n7nt\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.556133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.556080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d323113d-b1bc-44d2-9eec-68bb234bd7ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.656940 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.656845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d323113d-b1bc-44d2-9eec-68bb234bd7ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.656940 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.656882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d323113d-b1bc-44d2-9eec-68bb234bd7ec-manifests\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.656940 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.656908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d323113d-b1bc-44d2-9eec-68bb234bd7ec-tmp\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.656940 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.656936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d323113d-b1bc-44d2-9eec-68bb234bd7ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.657254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.656955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7nt\" (UniqueName: \"kubernetes.io/projected/d323113d-b1bc-44d2-9eec-68bb234bd7ec-kube-api-access-6n7nt\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.657354 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.657328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d323113d-b1bc-44d2-9eec-68bb234bd7ec-manifests\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.659340 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.659315    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d323113d-b1bc-44d2-9eec-68bb234bd7ec-tmp\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.659449 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.659386    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d323113d-b1bc-44d2-9eec-68bb234bd7ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.659507 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.659479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d323113d-b1bc-44d2-9eec-68bb234bd7ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.666267 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.666244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7nt\" (UniqueName: \"kubernetes.io/projected/d323113d-b1bc-44d2-9eec-68bb234bd7ec-kube-api-access-6n7nt\") pod \"opendatahub-operator-controller-manager-685bc94585-kqxdj\" (UID: \"d323113d-b1bc-44d2-9eec-68bb234bd7ec\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.731380 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.731348    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:23.812395 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.812355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:23.832086 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.832063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:23.930266 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.930186    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:23.942913 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.942819    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"]
Jul 28 20:05:23.945697 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:05:23.945663    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd323113d_b1bc_44d2_9eec_68bb234bd7ec.slice/crio-8b12502fa557925cc9fb3d974983d385f1f5ab419d8a3f59c4116d12b24aba87 WatchSource:0}: Error finding container 8b12502fa557925cc9fb3d974983d385f1f5ab419d8a3f59c4116d12b24aba87: Status 404 returned error can't find the container with id 8b12502fa557925cc9fb3d974983d385f1f5ab419d8a3f59c4116d12b24aba87
Jul 28 20:05:23.947401 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.947383    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:05:23.974846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:23.974818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj" event={"ID":"d323113d-b1bc-44d2-9eec-68bb234bd7ec","Type":"ContainerStarted","Data":"8b12502fa557925cc9fb3d974983d385f1f5ab419d8a3f59c4116d12b24aba87"}
Jul 28 20:05:24.015814 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:24.015782    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:24.076125 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:24.076094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:24.213069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:24.212990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:26.988595 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:26.988562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj" event={"ID":"d323113d-b1bc-44d2-9eec-68bb234bd7ec","Type":"ContainerStarted","Data":"51f6441d5749ab737c1b40ae5fa6380801536c9d5397dbe88100bd7429b92d5e"}
Jul 28 20:05:27.993209 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:27.993180    2582 generic.go:356] "Generic (PLEG): container finished" podID="d323113d-b1bc-44d2-9eec-68bb234bd7ec" containerID="51f6441d5749ab737c1b40ae5fa6380801536c9d5397dbe88100bd7429b92d5e" exitCode=0
Jul 28 20:05:27.993624 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:27.993234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj" event={"ID":"d323113d-b1bc-44d2-9eec-68bb234bd7ec","Type":"ContainerDied","Data":"51f6441d5749ab737c1b40ae5fa6380801536c9d5397dbe88100bd7429b92d5e"}
Jul 28 20:05:28.998630 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:28.998593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj" event={"ID":"d323113d-b1bc-44d2-9eec-68bb234bd7ec","Type":"ContainerStarted","Data":"ce2d248e824bd7797e7b4d0ce1cba67acfb5ffff352a003e2777cbe289b9fbff"}
Jul 28 20:05:28.999000 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:28.998712    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:29.019340 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:29.019273    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj" podStartSLOduration=1.7001175819999998 podStartE2EDuration="6.019257825s" totalImagesPullingTime="3.295449917s" totalInitContainerRuntime="1.023690326s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:29.017458153 +0000 UTC m=+378.871769219" watchObservedRunningTime="2026-07-28 20:05:29.019257825 +0000 UTC m=+378.873568894"
Jul 28 20:05:40.006697 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:40.006667    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-kqxdj"
Jul 28 20:05:43.419908 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:43.419858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:43.545976 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:43.545948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:54.239593 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.239517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:54.378762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.378725    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"]
Jul 28 20:05:54.389737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.389677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.392815 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.392781    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 20:05:54.393259 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.393237    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 20:05:54.394067 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.394045    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 20:05:54.394315 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.394279    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 20:05:54.394657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.394635    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-s52fn\""
Jul 28 20:05:54.396605 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.396582    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"]
Jul 28 20:05:54.497729 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.497642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7z74\" (UniqueName: \"kubernetes.io/projected/9f135a0a-1998-4886-94b9-934315a388c6-kube-api-access-w7z74\") pod \"kube-auth-proxy-6cdf7b948d-9lqhz\" (UID: \"9f135a0a-1998-4886-94b9-934315a388c6\") " pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.497729 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.497683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f135a0a-1998-4886-94b9-934315a388c6-tmp\") pod \"kube-auth-proxy-6cdf7b948d-9lqhz\" (UID: \"9f135a0a-1998-4886-94b9-934315a388c6\") " pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.497729 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.497708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f135a0a-1998-4886-94b9-934315a388c6-tls-certs\") pod \"kube-auth-proxy-6cdf7b948d-9lqhz\" (UID: \"9f135a0a-1998-4886-94b9-934315a388c6\") " pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.566977 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.566941    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:54.598583 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.598553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7z74\" (UniqueName: \"kubernetes.io/projected/9f135a0a-1998-4886-94b9-934315a388c6-kube-api-access-w7z74\") pod \"kube-auth-proxy-6cdf7b948d-9lqhz\" (UID: \"9f135a0a-1998-4886-94b9-934315a388c6\") " pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.598583 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.598585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f135a0a-1998-4886-94b9-934315a388c6-tmp\") pod \"kube-auth-proxy-6cdf7b948d-9lqhz\" (UID: \"9f135a0a-1998-4886-94b9-934315a388c6\") " pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.598789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.598609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f135a0a-1998-4886-94b9-934315a388c6-tls-certs\") pod \"kube-auth-proxy-6cdf7b948d-9lqhz\" (UID: \"9f135a0a-1998-4886-94b9-934315a388c6\") " pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.600952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.600920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f135a0a-1998-4886-94b9-934315a388c6-tmp\") pod \"kube-auth-proxy-6cdf7b948d-9lqhz\" (UID: \"9f135a0a-1998-4886-94b9-934315a388c6\") " pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.601200 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.601179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f135a0a-1998-4886-94b9-934315a388c6-tls-certs\") pod \"kube-auth-proxy-6cdf7b948d-9lqhz\" (UID: \"9f135a0a-1998-4886-94b9-934315a388c6\") " pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.608870 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.608838    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7z74\" (UniqueName: \"kubernetes.io/projected/9f135a0a-1998-4886-94b9-934315a388c6-kube-api-access-w7z74\") pod \"kube-auth-proxy-6cdf7b948d-9lqhz\" (UID: \"9f135a0a-1998-4886-94b9-934315a388c6\") " pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.613104 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.613082    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fdbnv"]
Jul 28 20:05:54.616414 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.616396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:05:54.619007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.618971    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 28 20:05:54.619107 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.619093    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-75cc9\""
Jul 28 20:05:54.626353 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.626325    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fdbnv"]
Jul 28 20:05:54.693427 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.693374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:54.699453 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.699433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3328a3b-22dc-490a-a299-3c15ad6ed2ca-cert\") pod \"odh-model-controller-7998c67dc4-fdbnv\" (UID: \"a3328a3b-22dc-490a-a299-3c15ad6ed2ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:05:54.699529 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.699472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xmc\" (UniqueName: \"kubernetes.io/projected/a3328a3b-22dc-490a-a299-3c15ad6ed2ca-kube-api-access-q5xmc\") pod \"odh-model-controller-7998c67dc4-fdbnv\" (UID: \"a3328a3b-22dc-490a-a299-3c15ad6ed2ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:05:54.701376 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.701361    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"
Jul 28 20:05:54.798762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.798731    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:54.800010 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.799984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3328a3b-22dc-490a-a299-3c15ad6ed2ca-cert\") pod \"odh-model-controller-7998c67dc4-fdbnv\" (UID: \"a3328a3b-22dc-490a-a299-3c15ad6ed2ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:05:54.800117 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.800040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xmc\" (UniqueName: \"kubernetes.io/projected/a3328a3b-22dc-490a-a299-3c15ad6ed2ca-kube-api-access-q5xmc\") pod \"odh-model-controller-7998c67dc4-fdbnv\" (UID: \"a3328a3b-22dc-490a-a299-3c15ad6ed2ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:05:54.800181 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:05:54.800137    2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 20:05:54.800240 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:05:54.800210    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3328a3b-22dc-490a-a299-3c15ad6ed2ca-cert podName:a3328a3b-22dc-490a-a299-3c15ad6ed2ca nodeName:}" failed. No retries permitted until 2026-07-28 20:05:55.300188961 +0000 UTC m=+405.154500010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a3328a3b-22dc-490a-a299-3c15ad6ed2ca-cert") pod "odh-model-controller-7998c67dc4-fdbnv" (UID: "a3328a3b-22dc-490a-a299-3c15ad6ed2ca") : secret "odh-model-controller-webhook-cert" not found
Jul 28 20:05:54.810509 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.810484    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xmc\" (UniqueName: \"kubernetes.io/projected/a3328a3b-22dc-490a-a299-3c15ad6ed2ca-kube-api-access-q5xmc\") pod \"odh-model-controller-7998c67dc4-fdbnv\" (UID: \"a3328a3b-22dc-490a-a299-3c15ad6ed2ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:05:54.847694 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.847664    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz"]
Jul 28 20:05:54.850408 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:05:54.850378    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f135a0a_1998_4886_94b9_934315a388c6.slice/crio-7f5d0d346ec13c271a0681c339da4c53d7f83ac40a4678c0e2994193a2ed0b4c WatchSource:0}: Error finding container 7f5d0d346ec13c271a0681c339da4c53d7f83ac40a4678c0e2994193a2ed0b4c: Status 404 returned error can't find the container with id 7f5d0d346ec13c271a0681c339da4c53d7f83ac40a4678c0e2994193a2ed0b4c
Jul 28 20:05:54.898886 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:54.898857    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:55.083922 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.083834    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:55.087531 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.087500    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz" event={"ID":"9f135a0a-1998-4886-94b9-934315a388c6","Type":"ContainerStarted","Data":"7f5d0d346ec13c271a0681c339da4c53d7f83ac40a4678c0e2994193a2ed0b4c"}
Jul 28 20:05:55.304772 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.304732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3328a3b-22dc-490a-a299-3c15ad6ed2ca-cert\") pod \"odh-model-controller-7998c67dc4-fdbnv\" (UID: \"a3328a3b-22dc-490a-a299-3c15ad6ed2ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:05:55.307145 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.307113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3328a3b-22dc-490a-a299-3c15ad6ed2ca-cert\") pod \"odh-model-controller-7998c67dc4-fdbnv\" (UID: \"a3328a3b-22dc-490a-a299-3c15ad6ed2ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:05:55.502606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.502574    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xl6q8"]
Jul 28 20:05:55.506123 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.506094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:05:55.506698 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.506664    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"]
Jul 28 20:05:55.508617 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.508594    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rhr82\""
Jul 28 20:05:55.508828 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.508810    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 28 20:05:55.510061 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.510041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.512269 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.512247    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 20:05:55.512415 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.512394    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:55.512482 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.512401    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 20:05:55.512482 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.512446    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 20:05:55.512802 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.512784    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 20:05:55.512893 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.512834    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-njwkk\""
Jul 28 20:05:55.517034 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.516670    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xl6q8"]
Jul 28 20:05:55.524045 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.524020    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"]
Jul 28 20:05:55.528273 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.528219    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:05:55.603228 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.603148    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:55.606958 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.606826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62rc4\" (UniqueName: \"kubernetes.io/projected/ea9236c2-355d-4eb3-9b21-0234a74bd117-kube-api-access-62rc4\") pod \"model-serving-api-54d89bd5fc-xl6q8\" (UID: \"ea9236c2-355d-4eb3-9b21-0234a74bd117\") " pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:05:55.606958 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.606870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.606958 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.606899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.607193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.606970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9236c2-355d-4eb3-9b21-0234a74bd117-tls-certs\") pod \"model-serving-api-54d89bd5fc-xl6q8\" (UID: \"ea9236c2-355d-4eb3-9b21-0234a74bd117\") " pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:05:55.607193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.607037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428fff8-711d-4a6c-aaae-47023cc40c39-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.607193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.607065    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.607193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.607122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.607193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.607149    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.607193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.607189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbm2\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-kube-api-access-cvbm2\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.697671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.697633    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fdbnv"]
Jul 28 20:05:55.701232 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:05:55.701203    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3328a3b_22dc_490a_a299_3c15ad6ed2ca.slice/crio-87eda837c129f4da074e8e505a0ebbd8ccacf1f7b0d295bd4b505543ec0b4434 WatchSource:0}: Error finding container 87eda837c129f4da074e8e505a0ebbd8ccacf1f7b0d295bd4b505543ec0b4434: Status 404 returned error can't find the container with id 87eda837c129f4da074e8e505a0ebbd8ccacf1f7b0d295bd4b505543ec0b4434
Jul 28 20:05:55.707929 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.707894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62rc4\" (UniqueName: \"kubernetes.io/projected/ea9236c2-355d-4eb3-9b21-0234a74bd117-kube-api-access-62rc4\") pod \"model-serving-api-54d89bd5fc-xl6q8\" (UID: \"ea9236c2-355d-4eb3-9b21-0234a74bd117\") " pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:05:55.708051 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.707939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.708051 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.707992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.708161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.708063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9236c2-355d-4eb3-9b21-0234a74bd117-tls-certs\") pod \"model-serving-api-54d89bd5fc-xl6q8\" (UID: \"ea9236c2-355d-4eb3-9b21-0234a74bd117\") " pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:05:55.708161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.708096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428fff8-711d-4a6c-aaae-47023cc40c39-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.708161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.708123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.708391 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.708184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.708391 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.708210    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.708391 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.708250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbm2\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-kube-api-access-cvbm2\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.709108 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.708757    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.709108 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:05:55.708863    2582 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 20:05:55.709108 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:05:55.708940    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea9236c2-355d-4eb3-9b21-0234a74bd117-tls-certs podName:ea9236c2-355d-4eb3-9b21-0234a74bd117 nodeName:}" failed. No retries permitted until 2026-07-28 20:05:56.208919804 +0000 UTC m=+406.063230848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ea9236c2-355d-4eb3-9b21-0234a74bd117-tls-certs") pod "model-serving-api-54d89bd5fc-xl6q8" (UID: "ea9236c2-355d-4eb3-9b21-0234a74bd117") : secret "model-serving-api-tls" not found
Jul 28 20:05:55.710936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.710912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.712310 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.712253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.712891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.712773    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428fff8-711d-4a6c-aaae-47023cc40c39-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.712891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.712860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.717718 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.717669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbm2\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-kube-api-access-cvbm2\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.717863 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.717840    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-j4bq8\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.718009 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.717982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62rc4\" (UniqueName: \"kubernetes.io/projected/ea9236c2-355d-4eb3-9b21-0234a74bd117-kube-api-access-62rc4\") pod \"model-serving-api-54d89bd5fc-xl6q8\" (UID: \"ea9236c2-355d-4eb3-9b21-0234a74bd117\") " pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:05:55.775241 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.775167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:55.833520 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.833480    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:05:55.992728 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:55.992641    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"]
Jul 28 20:05:55.994817 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:05:55.994790    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf428fff8_711d_4a6c_aaae_47023cc40c39.slice/crio-0e7d8646521fb95f2273470a692def6dde6b3f5e1d0721ffcc5be4e4ce8ec8dd WatchSource:0}: Error finding container 0e7d8646521fb95f2273470a692def6dde6b3f5e1d0721ffcc5be4e4ce8ec8dd: Status 404 returned error can't find the container with id 0e7d8646521fb95f2273470a692def6dde6b3f5e1d0721ffcc5be4e4ce8ec8dd
Jul 28 20:05:56.092557 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:56.092458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv" event={"ID":"a3328a3b-22dc-490a-a299-3c15ad6ed2ca","Type":"ContainerStarted","Data":"87eda837c129f4da074e8e505a0ebbd8ccacf1f7b0d295bd4b505543ec0b4434"}
Jul 28 20:05:56.093565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:56.093538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8" event={"ID":"f428fff8-711d-4a6c-aaae-47023cc40c39","Type":"ContainerStarted","Data":"0e7d8646521fb95f2273470a692def6dde6b3f5e1d0721ffcc5be4e4ce8ec8dd"}
Jul 28 20:05:56.213467 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:56.213420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9236c2-355d-4eb3-9b21-0234a74bd117-tls-certs\") pod \"model-serving-api-54d89bd5fc-xl6q8\" (UID: \"ea9236c2-355d-4eb3-9b21-0234a74bd117\") " pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:05:56.216217 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:56.216191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9236c2-355d-4eb3-9b21-0234a74bd117-tls-certs\") pod \"model-serving-api-54d89bd5fc-xl6q8\" (UID: \"ea9236c2-355d-4eb3-9b21-0234a74bd117\") " pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:05:56.426365 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:56.426263    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:05:56.487572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:56.487535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:56.608819 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:56.608577    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xl6q8"]
Jul 28 20:05:56.613042 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:05:56.613012    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9236c2_355d_4eb3_9b21_0234a74bd117.slice/crio-e0361862677c334230b002c5a5a3eeafbd159cc7fddb6e9b76bbb948f4ab795c WatchSource:0}: Error finding container e0361862677c334230b002c5a5a3eeafbd159cc7fddb6e9b76bbb948f4ab795c: Status 404 returned error can't find the container with id e0361862677c334230b002c5a5a3eeafbd159cc7fddb6e9b76bbb948f4ab795c
Jul 28 20:05:57.099183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:05:57.099145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8" event={"ID":"ea9236c2-355d-4eb3-9b21-0234a74bd117","Type":"ContainerStarted","Data":"e0361862677c334230b002c5a5a3eeafbd159cc7fddb6e9b76bbb948f4ab795c"}
Jul 28 20:06:00.720627 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:00.720591    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:06:00.720909 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:00.720660    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:06:01.120650 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.120616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz" event={"ID":"9f135a0a-1998-4886-94b9-934315a388c6","Type":"ContainerStarted","Data":"6811933de4108cfe6a3abeb320888a051571865aedd2cd1737e813b461069eaf"}
Jul 28 20:06:01.122173 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.122146    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv" event={"ID":"a3328a3b-22dc-490a-a299-3c15ad6ed2ca","Type":"ContainerStarted","Data":"53f8710c3afe5dfc03a83c3c2fd4ecb2bf0c94bd7d28505da1910b40d4efacb9"}
Jul 28 20:06:01.122344 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.122206    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:06:01.123908 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.123826    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8" event={"ID":"f428fff8-711d-4a6c-aaae-47023cc40c39","Type":"ContainerStarted","Data":"cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd"}
Jul 28 20:06:01.124714 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.124243    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:06:01.126033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.125755    2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-j4bq8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:06:01.126033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.125809    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8" podUID="f428fff8-711d-4a6c-aaae-47023cc40c39" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:06:01.126033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.125994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8" event={"ID":"ea9236c2-355d-4eb3-9b21-0234a74bd117","Type":"ContainerStarted","Data":"9b88165c1f8217335f7dc27c69fe71fe925c5063389fba9da2ff8079043e8580"}
Jul 28 20:06:01.126249 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.126050    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:06:01.140063 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.139997    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6cdf7b948d-9lqhz" podStartSLOduration=1.337391693 podStartE2EDuration="7.139975994s" totalImagesPullingTime="5.802584301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:01.138438572 +0000 UTC m=+410.992749640" watchObservedRunningTime="2026-07-28 20:06:01.139975994 +0000 UTC m=+410.994287060"
Jul 28 20:06:01.154661 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.154603    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv" podStartSLOduration=2.1982065410000002 podStartE2EDuration="7.15458225s" totalImagesPullingTime="4.956375709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:01.154087704 +0000 UTC m=+411.008398774" watchObservedRunningTime="2026-07-28 20:06:01.15458225 +0000 UTC m=+411.008893315"
Jul 28 20:06:01.174367 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.174315    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8" podStartSLOduration=1.450898967 podStartE2EDuration="6.174276644s" totalImagesPullingTime="4.723377677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:01.173037526 +0000 UTC m=+411.027348593" watchObservedRunningTime="2026-07-28 20:06:01.174276644 +0000 UTC m=+411.028587711"
Jul 28 20:06:01.190129 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.190083    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8" podStartSLOduration=2.143118728 podStartE2EDuration="6.190067663s" totalImagesPullingTime="4.046948935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:01.189572233 +0000 UTC m=+411.043883319" watchObservedRunningTime="2026-07-28 20:06:01.190067663 +0000 UTC m=+411.044378728"
Jul 28 20:06:01.271340 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.271258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:01.374705 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.374613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:01.963960 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:01.963920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:02.080369 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:02.080338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:02.134492 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:02.134465    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:06:02.332785 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:02.332741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:05.558924 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.558892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:05.625340 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.625300    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"]
Jul 28 20:06:05.628916 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.628894    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:05.631662 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.631640    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 28 20:06:05.631786 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.631671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6k4v7\""
Jul 28 20:06:05.634908 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.634882    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"]
Jul 28 20:06:05.638401 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.638382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:05.641839 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.641816    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"]
Jul 28 20:06:05.642087 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.642066    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7khzx\""
Jul 28 20:06:05.642232 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.642208    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 28 20:06:05.652946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.652914    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"]
Jul 28 20:06:05.665970 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.665938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:05.744955 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.744923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:05.803374 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.803340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhzqh\" (UniqueName: \"kubernetes.io/projected/a4e8ff70-c912-489d-80f9-899857887c3b-kube-api-access-zhzqh\") pod \"llmisvc-controller-manager-777c6bd7d5-jh9kg\" (UID: \"a4e8ff70-c912-489d-80f9-899857887c3b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:05.803578 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.803389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4e8ff70-c912-489d-80f9-899857887c3b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jh9kg\" (UID: \"a4e8ff70-c912-489d-80f9-899857887c3b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:05.803578 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.803419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlqfj\" (UniqueName: \"kubernetes.io/projected/e947b8e8-7fe7-4950-b024-951a48546a01-kube-api-access-xlqfj\") pod \"kserve-controller-manager-d6f4b65b8-pfc69\" (UID: \"e947b8e8-7fe7-4950-b024-951a48546a01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:05.803578 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.803502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e947b8e8-7fe7-4950-b024-951a48546a01-cert\") pod \"kserve-controller-manager-d6f4b65b8-pfc69\" (UID: \"e947b8e8-7fe7-4950-b024-951a48546a01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:05.857629 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.857538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:05.904423 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.904384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4e8ff70-c912-489d-80f9-899857887c3b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jh9kg\" (UID: \"a4e8ff70-c912-489d-80f9-899857887c3b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:05.904600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.904443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlqfj\" (UniqueName: \"kubernetes.io/projected/e947b8e8-7fe7-4950-b024-951a48546a01-kube-api-access-xlqfj\") pod \"kserve-controller-manager-d6f4b65b8-pfc69\" (UID: \"e947b8e8-7fe7-4950-b024-951a48546a01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:05.904600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.904476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e947b8e8-7fe7-4950-b024-951a48546a01-cert\") pod \"kserve-controller-manager-d6f4b65b8-pfc69\" (UID: \"e947b8e8-7fe7-4950-b024-951a48546a01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:05.904600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.904532    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhzqh\" (UniqueName: \"kubernetes.io/projected/a4e8ff70-c912-489d-80f9-899857887c3b-kube-api-access-zhzqh\") pod \"llmisvc-controller-manager-777c6bd7d5-jh9kg\" (UID: \"a4e8ff70-c912-489d-80f9-899857887c3b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:05.906895 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.906867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e947b8e8-7fe7-4950-b024-951a48546a01-cert\") pod \"kserve-controller-manager-d6f4b65b8-pfc69\" (UID: \"e947b8e8-7fe7-4950-b024-951a48546a01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:05.907002 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.906945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4e8ff70-c912-489d-80f9-899857887c3b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jh9kg\" (UID: \"a4e8ff70-c912-489d-80f9-899857887c3b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:05.923584 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.923553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlqfj\" (UniqueName: \"kubernetes.io/projected/e947b8e8-7fe7-4950-b024-951a48546a01-kube-api-access-xlqfj\") pod \"kserve-controller-manager-d6f4b65b8-pfc69\" (UID: \"e947b8e8-7fe7-4950-b024-951a48546a01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:05.927316 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.927269    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhzqh\" (UniqueName: \"kubernetes.io/projected/a4e8ff70-c912-489d-80f9-899857887c3b-kube-api-access-zhzqh\") pod \"llmisvc-controller-manager-777c6bd7d5-jh9kg\" (UID: \"a4e8ff70-c912-489d-80f9-899857887c3b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:05.940176 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.940146    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:05.949266 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.949248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:05.995990 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:05.995955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:06.077844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:06.077820    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"]
Jul 28 20:06:06.080061 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:06:06.080036    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode947b8e8_7fe7_4950_b024_951a48546a01.slice/crio-fbd5bf5c3d8a5acdb25c69d0eb28d27c53cc4bfa5fe9c21d25f4a9bbef8f54da WatchSource:0}: Error finding container fbd5bf5c3d8a5acdb25c69d0eb28d27c53cc4bfa5fe9c21d25f4a9bbef8f54da: Status 404 returned error can't find the container with id fbd5bf5c3d8a5acdb25c69d0eb28d27c53cc4bfa5fe9c21d25f4a9bbef8f54da
Jul 28 20:06:06.099823 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:06.099796    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"]
Jul 28 20:06:06.102856 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:06:06.102831    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda4e8ff70_c912_489d_80f9_899857887c3b.slice/crio-1aa3faa6b9ac128fbae3f2943acdc6a1aca949bd034c9e6467995834aacd04cd WatchSource:0}: Error finding container 1aa3faa6b9ac128fbae3f2943acdc6a1aca949bd034c9e6467995834aacd04cd: Status 404 returned error can't find the container with id 1aa3faa6b9ac128fbae3f2943acdc6a1aca949bd034c9e6467995834aacd04cd
Jul 28 20:06:06.152197 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:06.152109    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg" event={"ID":"a4e8ff70-c912-489d-80f9-899857887c3b","Type":"ContainerStarted","Data":"1aa3faa6b9ac128fbae3f2943acdc6a1aca949bd034c9e6467995834aacd04cd"}
Jul 28 20:06:06.153196 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:06.153162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69" event={"ID":"e947b8e8-7fe7-4950-b024-951a48546a01","Type":"ContainerStarted","Data":"fbd5bf5c3d8a5acdb25c69d0eb28d27c53cc4bfa5fe9c21d25f4a9bbef8f54da"}
Jul 28 20:06:09.170101 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.170056    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69" event={"ID":"e947b8e8-7fe7-4950-b024-951a48546a01","Type":"ContainerStarted","Data":"d7d7453909e6d79ed984d26cb4b491b287ad1a8a154c82364827bf5d7a12ff77"}
Jul 28 20:06:09.170618 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.170154    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:09.181342 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.181312    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"]
Jul 28 20:06:09.181558 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.181529    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8" podUID="f428fff8-711d-4a6c-aaae-47023cc40c39" containerName="discovery" containerID="cri-o://cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd" gracePeriod=30
Jul 28 20:06:09.195447 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.195394    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69" podStartSLOduration=1.65421531 podStartE2EDuration="4.19537449s" totalImagesPullingTime="2.54115918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:09.194853247 +0000 UTC m=+419.049164334" watchObservedRunningTime="2026-07-28 20:06:09.19537449 +0000 UTC m=+419.049685557"
Jul 28 20:06:09.594074 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.594047    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:06:09.740155 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.740124    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-cacerts\") pod \"f428fff8-711d-4a6c-aaae-47023cc40c39\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") "
Jul 28 20:06:09.740339 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.740171    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-ca-configmap\") pod \"f428fff8-711d-4a6c-aaae-47023cc40c39\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") "
Jul 28 20:06:09.740339 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.740194    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvbm2\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-kube-api-access-cvbm2\") pod \"f428fff8-711d-4a6c-aaae-47023cc40c39\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") "
Jul 28 20:06:09.740339 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.740310    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-token\") pod \"f428fff8-711d-4a6c-aaae-47023cc40c39\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") "
Jul 28 20:06:09.740511 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.740382    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-dns-cert\") pod \"f428fff8-711d-4a6c-aaae-47023cc40c39\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") "
Jul 28 20:06:09.740511 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.740408    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428fff8-711d-4a6c-aaae-47023cc40c39-local-certs\") pod \"f428fff8-711d-4a6c-aaae-47023cc40c39\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") "
Jul 28 20:06:09.740619 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.740588    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f428fff8-711d-4a6c-aaae-47023cc40c39" (UID: "f428fff8-711d-4a6c-aaae-47023cc40c39"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:06:09.740729 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.740708    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-kubeconfig\") pod \"f428fff8-711d-4a6c-aaae-47023cc40c39\" (UID: \"f428fff8-711d-4a6c-aaae-47023cc40c39\") "
Jul 28 20:06:09.740970 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.740950    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-ca-configmap\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:06:09.742634 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.742603    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-token" (OuterVolumeSpecName: "istio-token") pod "f428fff8-711d-4a6c-aaae-47023cc40c39" (UID: "f428fff8-711d-4a6c-aaae-47023cc40c39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:06:09.742745 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.742613    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-kube-api-access-cvbm2" (OuterVolumeSpecName: "kube-api-access-cvbm2") pod "f428fff8-711d-4a6c-aaae-47023cc40c39" (UID: "f428fff8-711d-4a6c-aaae-47023cc40c39"). InnerVolumeSpecName "kube-api-access-cvbm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:06:09.742800 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.742771    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-cacerts" (OuterVolumeSpecName: "cacerts") pod "f428fff8-711d-4a6c-aaae-47023cc40c39" (UID: "f428fff8-711d-4a6c-aaae-47023cc40c39"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:06:09.742853 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.742839    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f428fff8-711d-4a6c-aaae-47023cc40c39-local-certs" (OuterVolumeSpecName: "local-certs") pod "f428fff8-711d-4a6c-aaae-47023cc40c39" (UID: "f428fff8-711d-4a6c-aaae-47023cc40c39"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:06:09.743104 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.743087    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f428fff8-711d-4a6c-aaae-47023cc40c39" (UID: "f428fff8-711d-4a6c-aaae-47023cc40c39"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:06:09.743142 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.743097    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f428fff8-711d-4a6c-aaae-47023cc40c39" (UID: "f428fff8-711d-4a6c-aaae-47023cc40c39"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:06:09.841394 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.841355    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-csr-dns-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:06:09.841394 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.841386    2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428fff8-711d-4a6c-aaae-47023cc40c39-local-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:06:09.841394 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.841395    2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-kubeconfig\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:06:09.841394 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.841403    2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f428fff8-711d-4a6c-aaae-47023cc40c39-cacerts\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:06:09.841652 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.841412    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvbm2\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-kube-api-access-cvbm2\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:06:09.841652 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:09.841429    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f428fff8-711d-4a6c-aaae-47023cc40c39-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:06:10.175341 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.175229    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg" event={"ID":"a4e8ff70-c912-489d-80f9-899857887c3b","Type":"ContainerStarted","Data":"d90aaa66d0e32d8d99cca0537bf7d06c30b7dccdb8d61533bb4d74ce201babdd"}
Jul 28 20:06:10.175774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.175452    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:10.176529 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.176509    2582 generic.go:356] "Generic (PLEG): container finished" podID="f428fff8-711d-4a6c-aaae-47023cc40c39" containerID="cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd" exitCode=0
Jul 28 20:06:10.176606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.176539    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8" event={"ID":"f428fff8-711d-4a6c-aaae-47023cc40c39","Type":"ContainerDied","Data":"cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd"}
Jul 28 20:06:10.176606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.176554    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"
Jul 28 20:06:10.176606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.176589    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8" event={"ID":"f428fff8-711d-4a6c-aaae-47023cc40c39","Type":"ContainerDied","Data":"0e7d8646521fb95f2273470a692def6dde6b3f5e1d0721ffcc5be4e4ce8ec8dd"}
Jul 28 20:06:10.176747 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.176619    2582 scope.go:117] "RemoveContainer" containerID="cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd"
Jul 28 20:06:10.185243 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.185225    2582 scope.go:117] "RemoveContainer" containerID="cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd"
Jul 28 20:06:10.185530 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:06:10.185513    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd\": container with ID starting with cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd not found: ID does not exist" containerID="cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd"
Jul 28 20:06:10.185584 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.185538    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd"} err="failed to get container status \"cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd\": rpc error: code = NotFound desc = could not find container \"cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd\": container with ID starting with cc035ed80c08d90da8cc819b26dc0e9a8a20dcb983bf0e40f2f0d7aa9a784fcd not found: ID does not exist"
Jul 28 20:06:10.192036 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.191987    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg" podStartSLOduration=1.7811022570000001 podStartE2EDuration="5.19197308s" totalImagesPullingTime="3.410870823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:10.190630057 +0000 UTC m=+420.044941124" watchObservedRunningTime="2026-07-28 20:06:10.19197308 +0000 UTC m=+420.046284146"
Jul 28 20:06:10.204665 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.204629    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"]
Jul 28 20:06:10.210630 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.210606    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-j4bq8"]
Jul 28 20:06:10.585911 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:10.585877    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f428fff8-711d-4a6c-aaae-47023cc40c39" path="/var/lib/kubelet/pods/f428fff8-711d-4a6c-aaae-47023cc40c39/volumes"
Jul 28 20:06:12.136267 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:12.136235    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fdbnv"
Jul 28 20:06:12.138175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:12.138152    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xl6q8"
Jul 28 20:06:40.182510 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:40.182433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfc69"
Jul 28 20:06:41.182950 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:41.182918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jh9kg"
Jul 28 20:06:43.456416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:43.456382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:43.548858 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:43.548830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:44.871846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:44.871811    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:44.992344 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:44.992306    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bkqss"]
Jul 28 20:06:44.992633 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:44.992621    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f428fff8-711d-4a6c-aaae-47023cc40c39" containerName="discovery"
Jul 28 20:06:44.992675 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:44.992634    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f428fff8-711d-4a6c-aaae-47023cc40c39" containerName="discovery"
Jul 28 20:06:44.992708 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:44.992681    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f428fff8-711d-4a6c-aaae-47023cc40c39" containerName="discovery"
Jul 28 20:06:44.994813 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:44.994797    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bkqss"
Jul 28 20:06:44.997902 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:44.997875    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 20:06:44.998098 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:44.997881    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-wxm2p\""
Jul 28 20:06:44.998820 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:44.998800    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 20:06:45.009083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.009054    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bkqss"]
Jul 28 20:06:45.101622 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.101585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm97h\" (UniqueName: \"kubernetes.io/projected/46e74188-be49-4c6b-a39c-4cddaa2180b0-kube-api-access-lm97h\") pod \"kuadrant-operator-catalog-bkqss\" (UID: \"46e74188-be49-4c6b-a39c-4cddaa2180b0\") " pod="kuadrant-system/kuadrant-operator-catalog-bkqss"
Jul 28 20:06:45.202920 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.202816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm97h\" (UniqueName: \"kubernetes.io/projected/46e74188-be49-4c6b-a39c-4cddaa2180b0-kube-api-access-lm97h\") pod \"kuadrant-operator-catalog-bkqss\" (UID: \"46e74188-be49-4c6b-a39c-4cddaa2180b0\") " pod="kuadrant-system/kuadrant-operator-catalog-bkqss"
Jul 28 20:06:45.217888 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.217847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm97h\" (UniqueName: \"kubernetes.io/projected/46e74188-be49-4c6b-a39c-4cddaa2180b0-kube-api-access-lm97h\") pod \"kuadrant-operator-catalog-bkqss\" (UID: \"46e74188-be49-4c6b-a39c-4cddaa2180b0\") " pod="kuadrant-system/kuadrant-operator-catalog-bkqss"
Jul 28 20:06:45.305561 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.305519    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bkqss"
Jul 28 20:06:45.356079 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.356041    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bkqss"]
Jul 28 20:06:45.434632 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.434603    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bkqss"]
Jul 28 20:06:45.437122 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:06:45.437094    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e74188_be49_4c6b_a39c_4cddaa2180b0.slice/crio-2f28b195df45c281dd74c09f36b0ab137ac34738bf8c9c2a8156690ee3488c6b WatchSource:0}: Error finding container 2f28b195df45c281dd74c09f36b0ab137ac34738bf8c9c2a8156690ee3488c6b: Status 404 returned error can't find the container with id 2f28b195df45c281dd74c09f36b0ab137ac34738bf8c9c2a8156690ee3488c6b
Jul 28 20:06:45.511680 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.511646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:45.564253 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.564220    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cznwc"]
Jul 28 20:06:45.567733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.567714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cznwc"
Jul 28 20:06:45.576492 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.576469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cznwc"]
Jul 28 20:06:45.578851 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.578802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:45.606573 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.606533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b562q\" (UniqueName: \"kubernetes.io/projected/9c2211be-6d12-4dcc-a703-70d3a2e3e047-kube-api-access-b562q\") pod \"kuadrant-operator-catalog-cznwc\" (UID: \"9c2211be-6d12-4dcc-a703-70d3a2e3e047\") " pod="kuadrant-system/kuadrant-operator-catalog-cznwc"
Jul 28 20:06:45.707642 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.707610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b562q\" (UniqueName: \"kubernetes.io/projected/9c2211be-6d12-4dcc-a703-70d3a2e3e047-kube-api-access-b562q\") pod \"kuadrant-operator-catalog-cznwc\" (UID: \"9c2211be-6d12-4dcc-a703-70d3a2e3e047\") " pod="kuadrant-system/kuadrant-operator-catalog-cznwc"
Jul 28 20:06:45.716812 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.716782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b562q\" (UniqueName: \"kubernetes.io/projected/9c2211be-6d12-4dcc-a703-70d3a2e3e047-kube-api-access-b562q\") pod \"kuadrant-operator-catalog-cznwc\" (UID: \"9c2211be-6d12-4dcc-a703-70d3a2e3e047\") " pod="kuadrant-system/kuadrant-operator-catalog-cznwc"
Jul 28 20:06:45.790928 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.790791    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:45.877651 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:45.877620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cznwc"
Jul 28 20:06:46.003537 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:46.003501    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cznwc"]
Jul 28 20:06:46.005614 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:06:46.005588    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2211be_6d12_4dcc_a703_70d3a2e3e047.slice/crio-b48890b7eb05078aeb0545f31e01ec84daa3f06f4875f2ec72574b11f5c6476f WatchSource:0}: Error finding container b48890b7eb05078aeb0545f31e01ec84daa3f06f4875f2ec72574b11f5c6476f: Status 404 returned error can't find the container with id b48890b7eb05078aeb0545f31e01ec84daa3f06f4875f2ec72574b11f5c6476f
Jul 28 20:06:46.007656 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:46.007633    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:46.325082 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:46.325049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cznwc" event={"ID":"9c2211be-6d12-4dcc-a703-70d3a2e3e047","Type":"ContainerStarted","Data":"b48890b7eb05078aeb0545f31e01ec84daa3f06f4875f2ec72574b11f5c6476f"}
Jul 28 20:06:46.326225 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:46.326197    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bkqss" event={"ID":"46e74188-be49-4c6b-a39c-4cddaa2180b0","Type":"ContainerStarted","Data":"2f28b195df45c281dd74c09f36b0ab137ac34738bf8c9c2a8156690ee3488c6b"}
Jul 28 20:06:48.337805 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:48.337765    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cznwc" event={"ID":"9c2211be-6d12-4dcc-a703-70d3a2e3e047","Type":"ContainerStarted","Data":"bc829da31d76411795840c407fa3dc95782f207a67bd8b10706a2312354d1905"}
Jul 28 20:06:48.339058 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:48.339035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bkqss" event={"ID":"46e74188-be49-4c6b-a39c-4cddaa2180b0","Type":"ContainerStarted","Data":"ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276"}
Jul 28 20:06:48.339169 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:48.339131    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-bkqss" podUID="46e74188-be49-4c6b-a39c-4cddaa2180b0" containerName="registry-server" containerID="cri-o://ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276" gracePeriod=2
Jul 28 20:06:48.353740 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:48.353458    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-cznwc" podStartSLOduration=1.241825768 podStartE2EDuration="3.353438464s" totalImagesPullingTime="2.111612696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:48.352716602 +0000 UTC m=+458.207027669" watchObservedRunningTime="2026-07-28 20:06:48.353438464 +0000 UTC m=+458.207749531"
Jul 28 20:06:48.368038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:48.367961    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bkqss" podStartSLOduration=1.684182053 podStartE2EDuration="4.367948275s" totalImagesPullingTime="2.683766222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:48.367240738 +0000 UTC m=+458.221551819" watchObservedRunningTime="2026-07-28 20:06:48.367948275 +0000 UTC m=+458.222259338"
Jul 28 20:06:48.587660 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:48.587634    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bkqss"
Jul 28 20:06:48.635313 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:48.635209    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm97h\" (UniqueName: \"kubernetes.io/projected/46e74188-be49-4c6b-a39c-4cddaa2180b0-kube-api-access-lm97h\") pod \"46e74188-be49-4c6b-a39c-4cddaa2180b0\" (UID: \"46e74188-be49-4c6b-a39c-4cddaa2180b0\") "
Jul 28 20:06:48.637437 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:48.637412    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e74188-be49-4c6b-a39c-4cddaa2180b0-kube-api-access-lm97h" (OuterVolumeSpecName: "kube-api-access-lm97h") pod "46e74188-be49-4c6b-a39c-4cddaa2180b0" (UID: "46e74188-be49-4c6b-a39c-4cddaa2180b0"). InnerVolumeSpecName "kube-api-access-lm97h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:06:48.736746 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:48.736705    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm97h\" (UniqueName: \"kubernetes.io/projected/46e74188-be49-4c6b-a39c-4cddaa2180b0-kube-api-access-lm97h\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:06:49.344912 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:49.344871    2582 generic.go:356] "Generic (PLEG): container finished" podID="46e74188-be49-4c6b-a39c-4cddaa2180b0" containerID="ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276" exitCode=0
Jul 28 20:06:49.345414 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:49.344933    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bkqss"
Jul 28 20:06:49.345414 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:49.344963    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bkqss" event={"ID":"46e74188-be49-4c6b-a39c-4cddaa2180b0","Type":"ContainerDied","Data":"ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276"}
Jul 28 20:06:49.345414 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:49.345066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bkqss" event={"ID":"46e74188-be49-4c6b-a39c-4cddaa2180b0","Type":"ContainerDied","Data":"2f28b195df45c281dd74c09f36b0ab137ac34738bf8c9c2a8156690ee3488c6b"}
Jul 28 20:06:49.345414 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:49.345099    2582 scope.go:117] "RemoveContainer" containerID="ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276"
Jul 28 20:06:49.353662 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:49.353643    2582 scope.go:117] "RemoveContainer" containerID="ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276"
Jul 28 20:06:49.353918 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:06:49.353897    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276\": container with ID starting with ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276 not found: ID does not exist" containerID="ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276"
Jul 28 20:06:49.353980 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:49.353925    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276"} err="failed to get container status \"ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276\": rpc error: code = NotFound desc = could not find container \"ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276\": container with ID starting with ecd00d4c556a647d2d4fb8d47ec180351bc1e3e6df987b705d2b9c39cb6c8276 not found: ID does not exist"
Jul 28 20:06:49.364814 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:49.364791    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bkqss"]
Jul 28 20:06:49.368120 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:49.368099    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bkqss"]
Jul 28 20:06:50.583327 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:50.583281    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46e74188-be49-4c6b-a39c-4cddaa2180b0" path="/var/lib/kubelet/pods/46e74188-be49-4c6b-a39c-4cddaa2180b0/volumes"
Jul 28 20:06:55.878229 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:55.878186    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-cznwc"
Jul 28 20:06:55.878668 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:55.878284    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-cznwc"
Jul 28 20:06:55.899554 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:55.899523    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-cznwc"
Jul 28 20:06:56.395877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:06:56.395851    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-cznwc"
Jul 28 20:07:11.255692 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:11.255650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:11.344685 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:11.344651    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:11.443365 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:11.443314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:11.951924 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:11.951873    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:12.052994 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:12.052961    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:14.089058 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:14.089027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:16.741965 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.741934    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"]
Jul 28 20:07:16.742368 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.742275    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e74188-be49-4c6b-a39c-4cddaa2180b0" containerName="registry-server"
Jul 28 20:07:16.742368 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.742300    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e74188-be49-4c6b-a39c-4cddaa2180b0" containerName="registry-server"
Jul 28 20:07:16.742450 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.742373    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e74188-be49-4c6b-a39c-4cddaa2180b0" containerName="registry-server"
Jul 28 20:07:16.750214 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.750197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:16.752744 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.752724    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4sg66\""
Jul 28 20:07:16.760879 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.760850    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"]
Jul 28 20:07:16.862264 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.862232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkx6l\" (UniqueName: \"kubernetes.io/projected/66103b43-ca5e-491a-beef-4e20c9521db6-kube-api-access-gkx6l\") pod \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" (UID: \"66103b43-ca5e-491a-beef-4e20c9521db6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:16.963522 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.963494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkx6l\" (UniqueName: \"kubernetes.io/projected/66103b43-ca5e-491a-beef-4e20c9521db6-kube-api-access-gkx6l\") pod \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" (UID: \"66103b43-ca5e-491a-beef-4e20c9521db6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:16.977560 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:16.977524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkx6l\" (UniqueName: \"kubernetes.io/projected/66103b43-ca5e-491a-beef-4e20c9521db6-kube-api-access-gkx6l\") pod \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" (UID: \"66103b43-ca5e-491a-beef-4e20c9521db6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:17.059330 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:17.059226    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:17.196010 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:17.195984    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"]
Jul 28 20:07:17.198578 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:07:17.198552    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66103b43_ca5e_491a_beef_4e20c9521db6.slice/crio-355116d073d1247dfa1caa0e7d423e061f37c6169c178ccdf7725adaf06688fd WatchSource:0}: Error finding container 355116d073d1247dfa1caa0e7d423e061f37c6169c178ccdf7725adaf06688fd: Status 404 returned error can't find the container with id 355116d073d1247dfa1caa0e7d423e061f37c6169c178ccdf7725adaf06688fd
Jul 28 20:07:17.463763 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:17.463684    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh" event={"ID":"66103b43-ca5e-491a-beef-4e20c9521db6","Type":"ContainerStarted","Data":"355116d073d1247dfa1caa0e7d423e061f37c6169c178ccdf7725adaf06688fd"}
Jul 28 20:07:17.494375 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:17.494343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:18.381661 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:18.381633    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:19.474040 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:19.473992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh" event={"ID":"66103b43-ca5e-491a-beef-4e20c9521db6","Type":"ContainerStarted","Data":"f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae"}
Jul 28 20:07:19.474457 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:19.474131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:20.796147 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:20.796087    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh" podStartSLOduration=3.172191469 podStartE2EDuration="4.79607083s" totalImagesPullingTime="1.623879361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:19.492891581 +0000 UTC m=+489.347202644" watchObservedRunningTime="2026-07-28 20:07:20.79607083 +0000 UTC m=+490.650381896"
Jul 28 20:07:20.796825 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:20.796804    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"]
Jul 28 20:07:20.800419 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:20.800398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"
Jul 28 20:07:20.802963 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:20.802946    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cp7gk\""
Jul 28 20:07:20.811658 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:20.811634    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"]
Jul 28 20:07:20.896960 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:20.896927    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfztr\" (UniqueName: \"kubernetes.io/projected/31c351c7-2986-4ae1-b9d6-c97d7e95a9da-kube-api-access-gfztr\") pod \"authorino-operator-75cc8d5cb7-h5bzb\" (UID: \"31c351c7-2986-4ae1-b9d6-c97d7e95a9da\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"
Jul 28 20:07:20.998152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:20.998101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfztr\" (UniqueName: \"kubernetes.io/projected/31c351c7-2986-4ae1-b9d6-c97d7e95a9da-kube-api-access-gfztr\") pod \"authorino-operator-75cc8d5cb7-h5bzb\" (UID: \"31c351c7-2986-4ae1-b9d6-c97d7e95a9da\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"
Jul 28 20:07:21.015655 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:21.015627    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfztr\" (UniqueName: \"kubernetes.io/projected/31c351c7-2986-4ae1-b9d6-c97d7e95a9da-kube-api-access-gfztr\") pod \"authorino-operator-75cc8d5cb7-h5bzb\" (UID: \"31c351c7-2986-4ae1-b9d6-c97d7e95a9da\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"
Jul 28 20:07:21.109956 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:21.109865    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"
Jul 28 20:07:21.240095 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:21.240069    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"]
Jul 28 20:07:21.242712 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:07:21.242679    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c351c7_2986_4ae1_b9d6_c97d7e95a9da.slice/crio-a9b9f75a859da36f37cdb177315c0fef30999e2f2fe100aba36a41cd1df406f9 WatchSource:0}: Error finding container a9b9f75a859da36f37cdb177315c0fef30999e2f2fe100aba36a41cd1df406f9: Status 404 returned error can't find the container with id a9b9f75a859da36f37cdb177315c0fef30999e2f2fe100aba36a41cd1df406f9
Jul 28 20:07:21.350623 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:21.350591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:21.487201 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:21.487172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb" event={"ID":"31c351c7-2986-4ae1-b9d6-c97d7e95a9da","Type":"ContainerStarted","Data":"a9b9f75a859da36f37cdb177315c0fef30999e2f2fe100aba36a41cd1df406f9"}
Jul 28 20:07:21.515716 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:21.515680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:21.711197 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:21.711168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:21.906595 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:21.906517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:22.152007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:22.151974    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:23.952136 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:23.952056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:24.501125 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:24.501090    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb" event={"ID":"31c351c7-2986-4ae1-b9d6-c97d7e95a9da","Type":"ContainerStarted","Data":"a69e2b330b496629567982d5a6047edbd2f818abc01566fafd68d1fb3e109ae6"}
Jul 28 20:07:24.501325 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:24.501196    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"
Jul 28 20:07:24.519676 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:24.519625    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb" podStartSLOduration=2.272480464 podStartE2EDuration="4.519606832s" totalImagesPullingTime="2.247126368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:24.518447478 +0000 UTC m=+494.372758543" watchObservedRunningTime="2026-07-28 20:07:24.519606832 +0000 UTC m=+494.373917898"
Jul 28 20:07:25.735642 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.735608    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:25.819242 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.819207    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"]
Jul 28 20:07:25.822625 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.822610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:25.825407 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.825383    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jrtxx\""
Jul 28 20:07:25.836032 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.836003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4c1363-74e4-4591-b943-69fa02146f42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" (UID: \"4b4c1363-74e4-4591-b943-69fa02146f42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:25.836156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.836119    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnc8p\" (UniqueName: \"kubernetes.io/projected/4b4c1363-74e4-4591-b943-69fa02146f42-kube-api-access-wnc8p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" (UID: \"4b4c1363-74e4-4591-b943-69fa02146f42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:25.837661 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.837640    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"]
Jul 28 20:07:25.936663 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.936627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4c1363-74e4-4591-b943-69fa02146f42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" (UID: \"4b4c1363-74e4-4591-b943-69fa02146f42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:25.936832 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.936708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc8p\" (UniqueName: \"kubernetes.io/projected/4b4c1363-74e4-4591-b943-69fa02146f42-kube-api-access-wnc8p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" (UID: \"4b4c1363-74e4-4591-b943-69fa02146f42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:25.937004 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.936983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4c1363-74e4-4591-b943-69fa02146f42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" (UID: \"4b4c1363-74e4-4591-b943-69fa02146f42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:25.945792 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:25.945763    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc8p\" (UniqueName: \"kubernetes.io/projected/4b4c1363-74e4-4591-b943-69fa02146f42-kube-api-access-wnc8p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" (UID: \"4b4c1363-74e4-4591-b943-69fa02146f42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:26.132022 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:26.131937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:26.260244 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:26.260221    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"]
Jul 28 20:07:26.262281 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:07:26.262253    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4c1363_74e4_4591_b943_69fa02146f42.slice/crio-13fd63f30ab49c961db435696ceef088a5ba95ffc602d87404d2137976cc4ad9 WatchSource:0}: Error finding container 13fd63f30ab49c961db435696ceef088a5ba95ffc602d87404d2137976cc4ad9: Status 404 returned error can't find the container with id 13fd63f30ab49c961db435696ceef088a5ba95ffc602d87404d2137976cc4ad9
Jul 28 20:07:26.510691 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:26.510658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc" event={"ID":"4b4c1363-74e4-4591-b943-69fa02146f42","Type":"ContainerStarted","Data":"13fd63f30ab49c961db435696ceef088a5ba95ffc602d87404d2137976cc4ad9"}
Jul 28 20:07:26.968308 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:26.968194    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:28.949414 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:28.949382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:30.480938 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:30.480911    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:30.516863 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:30.516823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:30.618422 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:30.618383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:32.538031 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:32.537996    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc" event={"ID":"4b4c1363-74e4-4591-b943-69fa02146f42","Type":"ContainerStarted","Data":"14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3"}
Jul 28 20:07:32.538447 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:32.538082    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:32.557006 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:32.556963    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc" podStartSLOduration=1.7492298050000001 podStartE2EDuration="7.556948747s" totalImagesPullingTime="5.807718942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:32.554839826 +0000 UTC m=+502.409150893" watchObservedRunningTime="2026-07-28 20:07:32.556948747 +0000 UTC m=+502.411259813"
Jul 28 20:07:34.684161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:34.684128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:34.915576 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:34.915534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:35.011220 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:35.011186    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:35.070989 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:35.070953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:35.297668 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:35.297573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:35.398722 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:35.398690    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:35.508175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:35.508145    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h5bzb"
Jul 28 20:07:38.548844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:38.548807    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:38.650055 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:38.650022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:43.544526 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:43.544489    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:44.568610 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.568574    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"]
Jul 28 20:07:44.572319 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.572281    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:44.588324 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.588284    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"]
Jul 28 20:07:44.670568 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.670534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" (UID: \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:44.670737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.670575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27b8\" (UniqueName: \"kubernetes.io/projected/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-kube-api-access-k27b8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" (UID: \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:44.771463 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.771417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" (UID: \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:44.771660 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.771471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27b8\" (UniqueName: \"kubernetes.io/projected/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-kube-api-access-k27b8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" (UID: \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:44.771808 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.771785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" (UID: \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:44.785620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.785592    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27b8\" (UniqueName: \"kubernetes.io/projected/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-kube-api-access-k27b8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" (UID: \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:44.883198 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:44.883098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:45.012986 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.012943    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"]
Jul 28 20:07:45.015797 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:07:45.015763    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08029e5c_5d2a_4588_bafa_abd3bbb9a46e.slice/crio-0b73fe6d8ed6413e0a37a538a4f921f39593194639e1395970166554b8582e34 WatchSource:0}: Error finding container 0b73fe6d8ed6413e0a37a538a4f921f39593194639e1395970166554b8582e34: Status 404 returned error can't find the container with id 0b73fe6d8ed6413e0a37a538a4f921f39593194639e1395970166554b8582e34
Jul 28 20:07:45.203268 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.203178    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:45.213417 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.213386    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"]
Jul 28 20:07:45.217768 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.217743    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"]
Jul 28 20:07:45.229832 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.229804    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"]
Jul 28 20:07:45.230066 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.230033    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" containerName="manager" containerID="cri-o://14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3" gracePeriod=2
Jul 28 20:07:45.239000 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.238881    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"]
Jul 28 20:07:45.242883 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.242859    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"]
Jul 28 20:07:45.243172 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.243160    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" containerName="manager"
Jul 28 20:07:45.243231 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.243174    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" containerName="manager"
Jul 28 20:07:45.243231 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.243190    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" containerName="manager"
Jul 28 20:07:45.243231 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.243196    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" containerName="manager"
Jul 28 20:07:45.243350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.243272    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" containerName="manager"
Jul 28 20:07:45.243350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.243285    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" containerName="manager"
Jul 28 20:07:45.246049 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.246030    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:45.248186 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.248159    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.260834 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.260783    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"]
Jul 28 20:07:45.261037 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.261010    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" containerName="manager" containerID="cri-o://f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae" gracePeriod=2
Jul 28 20:07:45.262212 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.262192    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"]
Jul 28 20:07:45.265525 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.265504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:45.266448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.266418    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"]
Jul 28 20:07:45.275526 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.275501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157a59ae-f23f-4f1f-a1e6-df8773637486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-r9297\" (UID: \"157a59ae-f23f-4f1f-a1e6-df8773637486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:45.275649 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.275544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx7pd\" (UniqueName: \"kubernetes.io/projected/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-kube-api-access-xx7pd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zttsh\" (UID: \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:45.275649 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.275567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grw4\" (UniqueName: \"kubernetes.io/projected/157a59ae-f23f-4f1f-a1e6-df8773637486-kube-api-access-8grw4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-r9297\" (UID: \"157a59ae-f23f-4f1f-a1e6-df8773637486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:45.275762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.275650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zttsh\" (UID: \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:45.280759 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.280735    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"]
Jul 28 20:07:45.282778 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.282751    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"]
Jul 28 20:07:45.287641 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.287610    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.289666 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.289642    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:45.324433 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.324400    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:45.376984 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.376947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grw4\" (UniqueName: \"kubernetes.io/projected/157a59ae-f23f-4f1f-a1e6-df8773637486-kube-api-access-8grw4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-r9297\" (UID: \"157a59ae-f23f-4f1f-a1e6-df8773637486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:45.377123 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.377002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zttsh\" (UID: \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:45.377186 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.377155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157a59ae-f23f-4f1f-a1e6-df8773637486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-r9297\" (UID: \"157a59ae-f23f-4f1f-a1e6-df8773637486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:45.377242 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.377211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx7pd\" (UniqueName: \"kubernetes.io/projected/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-kube-api-access-xx7pd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zttsh\" (UID: \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:45.377432 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.377411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zttsh\" (UID: \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:45.377504 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.377487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157a59ae-f23f-4f1f-a1e6-df8773637486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-r9297\" (UID: \"157a59ae-f23f-4f1f-a1e6-df8773637486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:45.385879 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.385842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grw4\" (UniqueName: \"kubernetes.io/projected/157a59ae-f23f-4f1f-a1e6-df8773637486-kube-api-access-8grw4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-r9297\" (UID: \"157a59ae-f23f-4f1f-a1e6-df8773637486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:45.385983 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.385901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx7pd\" (UniqueName: \"kubernetes.io/projected/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-kube-api-access-xx7pd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zttsh\" (UID: \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:45.488973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.488946    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.491399 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.491359    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:45.493333 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.493278    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.494161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.494142    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:45.496297 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.496265    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:45.498153 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.498133    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.578457 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.578411    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnc8p\" (UniqueName: \"kubernetes.io/projected/4b4c1363-74e4-4591-b943-69fa02146f42-kube-api-access-wnc8p\") pod \"4b4c1363-74e4-4591-b943-69fa02146f42\" (UID: \"4b4c1363-74e4-4591-b943-69fa02146f42\") "
Jul 28 20:07:45.578961 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.578479    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4c1363-74e4-4591-b943-69fa02146f42-extensions-socket-volume\") pod \"4b4c1363-74e4-4591-b943-69fa02146f42\" (UID: \"4b4c1363-74e4-4591-b943-69fa02146f42\") "
Jul 28 20:07:45.578961 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.578512    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkx6l\" (UniqueName: \"kubernetes.io/projected/66103b43-ca5e-491a-beef-4e20c9521db6-kube-api-access-gkx6l\") pod \"66103b43-ca5e-491a-beef-4e20c9521db6\" (UID: \"66103b43-ca5e-491a-beef-4e20c9521db6\") "
Jul 28 20:07:45.578961 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.578898    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4c1363-74e4-4591-b943-69fa02146f42-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4b4c1363-74e4-4591-b943-69fa02146f42" (UID: "4b4c1363-74e4-4591-b943-69fa02146f42"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:07:45.580587 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.580566    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66103b43-ca5e-491a-beef-4e20c9521db6-kube-api-access-gkx6l" (OuterVolumeSpecName: "kube-api-access-gkx6l") pod "66103b43-ca5e-491a-beef-4e20c9521db6" (UID: "66103b43-ca5e-491a-beef-4e20c9521db6"). InnerVolumeSpecName "kube-api-access-gkx6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:07:45.580587 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.580580    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4c1363-74e4-4591-b943-69fa02146f42-kube-api-access-wnc8p" (OuterVolumeSpecName: "kube-api-access-wnc8p") pod "4b4c1363-74e4-4591-b943-69fa02146f42" (UID: "4b4c1363-74e4-4591-b943-69fa02146f42"). InnerVolumeSpecName "kube-api-access-wnc8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:07:45.600077 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.600045    2582 generic.go:356] "Generic (PLEG): container finished" podID="4b4c1363-74e4-4591-b943-69fa02146f42" containerID="14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3" exitCode=0
Jul 28 20:07:45.600200 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.600092    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.600200 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.600151    2582 scope.go:117] "RemoveContainer" containerID="14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3"
Jul 28 20:07:45.601484 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.601459    2582 generic.go:356] "Generic (PLEG): container finished" podID="66103b43-ca5e-491a-beef-4e20c9521db6" containerID="f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae" exitCode=0
Jul 28 20:07:45.601581 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.601509    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:45.602553 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.602524    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:45.603874 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.603855    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9" container="manager"
Jul 28 20:07:45.604337 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.604310    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.606243 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.606217    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.608697 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.608672    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:45.609205 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.609172    2582 scope.go:117] "RemoveContainer" containerID="14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3"
Jul 28 20:07:45.609501 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.609475    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3\": container with ID starting with 14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3 not found: ID does not exist" containerID="14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3"
Jul 28 20:07:45.609574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.609512    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3"} err="failed to get container status \"14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3\": rpc error: code = NotFound desc = could not find container \"14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3\": container with ID starting with 14a2eb7c71342c0f2e0a4feb6c1ad72f4e45fda757617b9f418c64bc1ec317f3 not found: ID does not exist"
Jul 28 20:07:45.609574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.609535    2582 scope.go:117] "RemoveContainer" containerID="f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae"
Jul 28 20:07:45.612873 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.612841    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:45.615116 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.615095    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:45.619820 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.619797    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:45.621172 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.621156    2582 scope.go:117] "RemoveContainer" containerID="f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae"
Jul 28 20:07:45.621480 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.621458    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae\": container with ID starting with f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae not found: ID does not exist" containerID="f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae"
Jul 28 20:07:45.621534 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.621492    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae"} err="failed to get container status \"f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae\": rpc error: code = NotFound desc = could not find container \"f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae\": container with ID starting with f8191eeef83433e6f66f5a2a623da5e7fcc80b6fa2ed1cf9fa250ea7d94ffaae not found: ID does not exist"
Jul 28 20:07:45.621790 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.621767    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.623736 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.623716    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rpjnh\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rpjnh"
Jul 28 20:07:45.625472 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.625454    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:45.627330 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:45.627312    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cdxjc\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdxjc"
Jul 28 20:07:45.648489 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.648454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:45.654126 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.654098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:45.680004 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.679968    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4c1363-74e4-4591-b943-69fa02146f42-extensions-socket-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:07:45.680004 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.680006    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkx6l\" (UniqueName: \"kubernetes.io/projected/66103b43-ca5e-491a-beef-4e20c9521db6-kube-api-access-gkx6l\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:07:45.680171 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.680024    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnc8p\" (UniqueName: \"kubernetes.io/projected/4b4c1363-74e4-4591-b943-69fa02146f42-kube-api-access-wnc8p\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:07:45.800880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.800840    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"]
Jul 28 20:07:45.804614 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:07:45.804577    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157a59ae_f23f_4f1f_a1e6_df8773637486.slice/crio-d346e57e3ecfc66c9423a496b4a9fdf52a5b6fc45d8354b4eee1a95545fc217c WatchSource:0}: Error finding container d346e57e3ecfc66c9423a496b4a9fdf52a5b6fc45d8354b4eee1a95545fc217c: Status 404 returned error can't find the container with id d346e57e3ecfc66c9423a496b4a9fdf52a5b6fc45d8354b4eee1a95545fc217c
Jul 28 20:07:45.824654 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:45.824631    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"]
Jul 28 20:07:45.826266 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:07:45.826239    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e29ce1c_cf25_45ea_8ca3_49d851f4cf38.slice/crio-09c64f4d0c49cf227ffe60ace6184a3e2351fd803411548b312b6f98cad4c477 WatchSource:0}: Error finding container 09c64f4d0c49cf227ffe60ace6184a3e2351fd803411548b312b6f98cad4c477: Status 404 returned error can't find the container with id 09c64f4d0c49cf227ffe60ace6184a3e2351fd803411548b312b6f98cad4c477
Jul 28 20:07:46.583356 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.583326    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4c1363-74e4-4591-b943-69fa02146f42" path="/var/lib/kubelet/pods/4b4c1363-74e4-4591-b943-69fa02146f42/volumes"
Jul 28 20:07:46.583713 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.583648    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" path="/var/lib/kubelet/pods/66103b43-ca5e-491a-beef-4e20c9521db6/volumes"
Jul 28 20:07:46.609022 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.608980    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" containerName="manager" containerID="cri-o://7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695" gracePeriod=2
Jul 28 20:07:46.610736 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.610710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh" event={"ID":"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38","Type":"ContainerStarted","Data":"d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3"}
Jul 28 20:07:46.610857 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.610763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh" event={"ID":"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38","Type":"ContainerStarted","Data":"09c64f4d0c49cf227ffe60ace6184a3e2351fd803411548b312b6f98cad4c477"}
Jul 28 20:07:46.610857 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.610794    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:46.612407 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.612385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297" event={"ID":"157a59ae-f23f-4f1f-a1e6-df8773637486","Type":"ContainerStarted","Data":"8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a"}
Jul 28 20:07:46.612526 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.612423    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297" event={"ID":"157a59ae-f23f-4f1f-a1e6-df8773637486","Type":"ContainerStarted","Data":"d346e57e3ecfc66c9423a496b4a9fdf52a5b6fc45d8354b4eee1a95545fc217c"}
Jul 28 20:07:46.612526 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.612464    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:46.638465 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.638423    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh" podStartSLOduration=1.6384081259999999 podStartE2EDuration="1.638408126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:46.638139567 +0000 UTC m=+516.492450665" watchObservedRunningTime="2026-07-28 20:07:46.638408126 +0000 UTC m=+516.492719251"
Jul 28 20:07:46.662709 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.662671    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297" podStartSLOduration=1.6626580930000001 podStartE2EDuration="1.662658093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:46.66048632 +0000 UTC m=+516.514797385" watchObservedRunningTime="2026-07-28 20:07:46.662658093 +0000 UTC m=+516.516969159"
Jul 28 20:07:46.853558 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.853537    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:46.856361 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:46.856336    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:46.888360 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.888336    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-extensions-socket-volume\") pod \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\" (UID: \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\") "
Jul 28 20:07:46.888461 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.888364    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k27b8\" (UniqueName: \"kubernetes.io/projected/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-kube-api-access-k27b8\") pod \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\" (UID: \"08029e5c-5d2a-4588-bafa-abd3bbb9a46e\") "
Jul 28 20:07:46.888626 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.888600    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "08029e5c-5d2a-4588-bafa-abd3bbb9a46e" (UID: "08029e5c-5d2a-4588-bafa-abd3bbb9a46e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:07:46.890338 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.890307    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-kube-api-access-k27b8" (OuterVolumeSpecName: "kube-api-access-k27b8") pod "08029e5c-5d2a-4588-bafa-abd3bbb9a46e" (UID: "08029e5c-5d2a-4588-bafa-abd3bbb9a46e"). InnerVolumeSpecName "kube-api-access-k27b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:07:46.989716 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.989686    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-extensions-socket-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:07:46.989716 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:46.989711    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27b8\" (UniqueName: \"kubernetes.io/projected/08029e5c-5d2a-4588-bafa-abd3bbb9a46e-kube-api-access-k27b8\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:07:47.619609 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:47.619576    2582 generic.go:356] "Generic (PLEG): container finished" podID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" containerID="7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695" exitCode=2
Jul 28 20:07:47.620066 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:47.619620    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:47.620066 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:47.619661    2582 scope.go:117] "RemoveContainer" containerID="7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695"
Jul 28 20:07:47.622066 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:47.622040    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:47.628121 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:47.628107    2582 scope.go:117] "RemoveContainer" containerID="7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695"
Jul 28 20:07:47.628378 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:47.628356    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695\": container with ID starting with 7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695 not found: ID does not exist" containerID="7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695"
Jul 28 20:07:47.628430 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:47.628386    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695"} err="failed to get container status \"7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695\": rpc error: code = NotFound desc = could not find container \"7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695\": container with ID starting with 7f5b75caf03d5adaec082b7ec41bd6eab20b8a6b4a413901ac7bee417339f695 not found: ID does not exist"
Jul 28 20:07:47.630036 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:47.630017    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-mpvp9\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mpvp9"
Jul 28 20:07:48.583124 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:48.583091    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08029e5c-5d2a-4588-bafa-abd3bbb9a46e" path="/var/lib/kubelet/pods/08029e5c-5d2a-4588-bafa-abd3bbb9a46e/volumes"
Jul 28 20:07:49.380759 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:49.380706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:49.481583 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:49.481552    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:53.191084 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:53.191026    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:53.250804 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:53.250771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:53.379388 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:53.379356    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:53.480233 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:53.480122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:57.621922 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.621894    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:57.622390 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.621972    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:07:57.695183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.695146    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"]
Jul 28 20:07:57.700113 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.695757    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh" podUID="4e29ce1c-cf25-45ea-8ca3-49d851f4cf38" containerName="manager" containerID="cri-o://d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3" gracePeriod=10
Jul 28 20:07:57.950091 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.950064    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:57.999325 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.999276    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"]
Jul 28 20:07:57.999658 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.999624    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e29ce1c-cf25-45ea-8ca3-49d851f4cf38" containerName="manager"
Jul 28 20:07:57.999720 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.999660    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e29ce1c-cf25-45ea-8ca3-49d851f4cf38" containerName="manager"
Jul 28 20:07:57.999720 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.999677    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" containerName="manager"
Jul 28 20:07:57.999720 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.999683    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" containerName="manager"
Jul 28 20:07:57.999809 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.999735    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e29ce1c-cf25-45ea-8ca3-49d851f4cf38" containerName="manager"
Jul 28 20:07:57.999809 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:57.999743    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="66103b43-ca5e-491a-beef-4e20c9521db6" containerName="manager"
Jul 28 20:07:58.005082 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.005056    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:07:58.015097 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.015069    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"]
Jul 28 20:07:58.074557 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.074522    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-extensions-socket-volume\") pod \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\" (UID: \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\") "
Jul 28 20:07:58.074557 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.074562    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx7pd\" (UniqueName: \"kubernetes.io/projected/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-kube-api-access-xx7pd\") pod \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\" (UID: \"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38\") "
Jul 28 20:07:58.074764 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.074682    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc94170-3847-4197-be1d-730c927c39ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-chtqt\" (UID: \"9fc94170-3847-4197-be1d-730c927c39ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:07:58.074764 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.074707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztph6\" (UniqueName: \"kubernetes.io/projected/9fc94170-3847-4197-be1d-730c927c39ae-kube-api-access-ztph6\") pod \"kuadrant-operator-controller-manager-86878c769f-chtqt\" (UID: \"9fc94170-3847-4197-be1d-730c927c39ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:07:58.074947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.074923    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4e29ce1c-cf25-45ea-8ca3-49d851f4cf38" (UID: "4e29ce1c-cf25-45ea-8ca3-49d851f4cf38"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:07:58.076633 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.076607    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-kube-api-access-xx7pd" (OuterVolumeSpecName: "kube-api-access-xx7pd") pod "4e29ce1c-cf25-45ea-8ca3-49d851f4cf38" (UID: "4e29ce1c-cf25-45ea-8ca3-49d851f4cf38"). InnerVolumeSpecName "kube-api-access-xx7pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:07:58.175389 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.175267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc94170-3847-4197-be1d-730c927c39ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-chtqt\" (UID: \"9fc94170-3847-4197-be1d-730c927c39ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:07:58.175389 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.175340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztph6\" (UniqueName: \"kubernetes.io/projected/9fc94170-3847-4197-be1d-730c927c39ae-kube-api-access-ztph6\") pod \"kuadrant-operator-controller-manager-86878c769f-chtqt\" (UID: \"9fc94170-3847-4197-be1d-730c927c39ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:07:58.175600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.175396    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-extensions-socket-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:07:58.175600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.175410    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx7pd\" (UniqueName: \"kubernetes.io/projected/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38-kube-api-access-xx7pd\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:07:58.175756 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.175732    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc94170-3847-4197-be1d-730c927c39ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-chtqt\" (UID: \"9fc94170-3847-4197-be1d-730c927c39ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:07:58.184574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.184551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztph6\" (UniqueName: \"kubernetes.io/projected/9fc94170-3847-4197-be1d-730c927c39ae-kube-api-access-ztph6\") pod \"kuadrant-operator-controller-manager-86878c769f-chtqt\" (UID: \"9fc94170-3847-4197-be1d-730c927c39ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:07:58.216077 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.216037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:58.309077 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.309044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:58.316685 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.316662    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:07:58.449738 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.449711    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"]
Jul 28 20:07:58.452148 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:07:58.452120    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc94170_3847_4197_be1d_730c927c39ae.slice/crio-f3c7eecd363faf47d505f0f26b1cecb0d779716808db39ee468c8849ed36091d WatchSource:0}: Error finding container f3c7eecd363faf47d505f0f26b1cecb0d779716808db39ee468c8849ed36091d: Status 404 returned error can't find the container with id f3c7eecd363faf47d505f0f26b1cecb0d779716808db39ee468c8849ed36091d
Jul 28 20:07:58.673082 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.673049    2582 generic.go:356] "Generic (PLEG): container finished" podID="4e29ce1c-cf25-45ea-8ca3-49d851f4cf38" containerID="d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3" exitCode=0
Jul 28 20:07:58.673565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.673093    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"
Jul 28 20:07:58.673565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.673135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh" event={"ID":"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38","Type":"ContainerDied","Data":"d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3"}
Jul 28 20:07:58.673565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.673192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh" event={"ID":"4e29ce1c-cf25-45ea-8ca3-49d851f4cf38","Type":"ContainerDied","Data":"09c64f4d0c49cf227ffe60ace6184a3e2351fd803411548b312b6f98cad4c477"}
Jul 28 20:07:58.673565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.673226    2582 scope.go:117] "RemoveContainer" containerID="d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3"
Jul 28 20:07:58.675001 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.674975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt" event={"ID":"9fc94170-3847-4197-be1d-730c927c39ae","Type":"ContainerStarted","Data":"fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013"}
Jul 28 20:07:58.675151 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.675030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt" event={"ID":"9fc94170-3847-4197-be1d-730c927c39ae","Type":"ContainerStarted","Data":"f3c7eecd363faf47d505f0f26b1cecb0d779716808db39ee468c8849ed36091d"}
Jul 28 20:07:58.675151 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.675062    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:07:58.685949 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.685922    2582 scope.go:117] "RemoveContainer" containerID="d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3"
Jul 28 20:07:58.686243 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:07:58.686220    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3\": container with ID starting with d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3 not found: ID does not exist" containerID="d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3"
Jul 28 20:07:58.686328 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.686253    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3"} err="failed to get container status \"d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3\": rpc error: code = NotFound desc = could not find container \"d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3\": container with ID starting with d7381859e60bb2f3040ab7da4cc339cb4f13b2c68baf378a6a1816f86bc860a3 not found: ID does not exist"
Jul 28 20:07:58.702960 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.702856    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt" podStartSLOduration=1.70284174 podStartE2EDuration="1.70284174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:58.701794312 +0000 UTC m=+528.556105379" watchObservedRunningTime="2026-07-28 20:07:58.70284174 +0000 UTC m=+528.557152806"
Jul 28 20:07:58.716915 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.716876    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"]
Jul 28 20:07:58.723418 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:07:58.723388    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zttsh"]
Jul 28 20:08:00.582460 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:00.582426    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e29ce1c-cf25-45ea-8ca3-49d851f4cf38" path="/var/lib/kubelet/pods/4e29ce1c-cf25-45ea-8ca3-49d851f4cf38/volumes"
Jul 28 20:08:03.924715 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:03.924630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:04.299839 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:04.299800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:09.682803 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:09.682771    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:08:09.734102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:09.734068    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"]
Jul 28 20:08:09.734369 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:09.734322    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297" podUID="157a59ae-f23f-4f1f-a1e6-df8773637486" containerName="manager" containerID="cri-o://8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a" gracePeriod=10
Jul 28 20:08:09.988633 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:09.988606    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:08:10.070260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.070220    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157a59ae-f23f-4f1f-a1e6-df8773637486-extensions-socket-volume\") pod \"157a59ae-f23f-4f1f-a1e6-df8773637486\" (UID: \"157a59ae-f23f-4f1f-a1e6-df8773637486\") "
Jul 28 20:08:10.070455 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.070274    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8grw4\" (UniqueName: \"kubernetes.io/projected/157a59ae-f23f-4f1f-a1e6-df8773637486-kube-api-access-8grw4\") pod \"157a59ae-f23f-4f1f-a1e6-df8773637486\" (UID: \"157a59ae-f23f-4f1f-a1e6-df8773637486\") "
Jul 28 20:08:10.070622 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.070598    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a59ae-f23f-4f1f-a1e6-df8773637486-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "157a59ae-f23f-4f1f-a1e6-df8773637486" (UID: "157a59ae-f23f-4f1f-a1e6-df8773637486"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:08:10.072300 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.072262    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157a59ae-f23f-4f1f-a1e6-df8773637486-kube-api-access-8grw4" (OuterVolumeSpecName: "kube-api-access-8grw4") pod "157a59ae-f23f-4f1f-a1e6-df8773637486" (UID: "157a59ae-f23f-4f1f-a1e6-df8773637486"). InnerVolumeSpecName "kube-api-access-8grw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:08:10.171549 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.171513    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157a59ae-f23f-4f1f-a1e6-df8773637486-extensions-socket-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:08:10.171549 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.171543    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8grw4\" (UniqueName: \"kubernetes.io/projected/157a59ae-f23f-4f1f-a1e6-df8773637486-kube-api-access-8grw4\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:08:10.404706 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.404672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:10.504949 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.504929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:10.729266 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.729163    2582 generic.go:356] "Generic (PLEG): container finished" podID="157a59ae-f23f-4f1f-a1e6-df8773637486" containerID="8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a" exitCode=0
Jul 28 20:08:10.729266 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.729219    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"
Jul 28 20:08:10.729266 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.729251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297" event={"ID":"157a59ae-f23f-4f1f-a1e6-df8773637486","Type":"ContainerDied","Data":"8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a"}
Jul 28 20:08:10.729837 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.729305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297" event={"ID":"157a59ae-f23f-4f1f-a1e6-df8773637486","Type":"ContainerDied","Data":"d346e57e3ecfc66c9423a496b4a9fdf52a5b6fc45d8354b4eee1a95545fc217c"}
Jul 28 20:08:10.729837 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.729331    2582 scope.go:117] "RemoveContainer" containerID="8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a"
Jul 28 20:08:10.737648 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.737484    2582 scope.go:117] "RemoveContainer" containerID="8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a"
Jul 28 20:08:10.737776 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:08:10.737753    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a\": container with ID starting with 8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a not found: ID does not exist" containerID="8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a"
Jul 28 20:08:10.737875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.737780    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a"} err="failed to get container status \"8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a\": rpc error: code = NotFound desc = could not find container \"8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a\": container with ID starting with 8f3d181e8d594475db8922a811aff90900c527412be8b6c11e29c540bcc0e35a not found: ID does not exist"
Jul 28 20:08:10.750349 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.750321    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"]
Jul 28 20:08:10.759719 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:10.759691    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-r9297"]
Jul 28 20:08:12.582540 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:12.582500    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="157a59ae-f23f-4f1f-a1e6-df8773637486" path="/var/lib/kubelet/pods/157a59ae-f23f-4f1f-a1e6-df8773637486/volumes"
Jul 28 20:08:13.974581 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:13.974548    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"]
Jul 28 20:08:13.975020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:13.975005    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157a59ae-f23f-4f1f-a1e6-df8773637486" containerName="manager"
Jul 28 20:08:13.975064 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:13.975023    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="157a59ae-f23f-4f1f-a1e6-df8773637486" containerName="manager"
Jul 28 20:08:13.975108 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:13.975097    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="157a59ae-f23f-4f1f-a1e6-df8773637486" containerName="manager"
Jul 28 20:08:13.979274 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:13.979257    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:13.981850 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:13.981829    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 20:08:13.981981 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:13.981833    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jzp4g\""
Jul 28 20:08:13.988901 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:13.988876    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"]
Jul 28 20:08:14.006447 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.006422    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:14.103841 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.103807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.103841 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.103842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcmb5\" (UniqueName: \"kubernetes.io/projected/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-kube-api-access-bcmb5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.104038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.103877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.104038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.103893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.104038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.103952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.104038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.103972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.104038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.103989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.104038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.104015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.104276 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.104104    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcmb5\" (UniqueName: \"kubernetes.io/projected/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-kube-api-access-bcmb5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205790 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205633    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205790 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205687    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205896 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.205949 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.205920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.206187 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.206166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.207849 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.207818    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.208069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.208048    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.213465 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.213445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.213775 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.213751    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcmb5\" (UniqueName: \"kubernetes.io/projected/2d099fda-4a87-45cb-a88f-a1d546cf1e6e-kube-api-access-bcmb5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-c5q2z\" (UID: \"2d099fda-4a87-45cb-a88f-a1d546cf1e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.290634 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.290607    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:14.379759 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.379710    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:14.433427 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.433406    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"]
Jul 28 20:08:14.435156 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:08:14.435083    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d099fda_4a87_45cb_a88f_a1d546cf1e6e.slice/crio-de5bed8633169526de518f5e41ab59d9f45ca904e6be8b5915922f2b1f568b0a WatchSource:0}: Error finding container de5bed8633169526de518f5e41ab59d9f45ca904e6be8b5915922f2b1f568b0a: Status 404 returned error can't find the container with id de5bed8633169526de518f5e41ab59d9f45ca904e6be8b5915922f2b1f568b0a
Jul 28 20:08:14.747389 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:14.747360    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z" event={"ID":"2d099fda-4a87-45cb-a88f-a1d546cf1e6e","Type":"ContainerStarted","Data":"de5bed8633169526de518f5e41ab59d9f45ca904e6be8b5915922f2b1f568b0a"}
Jul 28 20:08:32.158639 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:32.158609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:34.047121 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:34.047063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:46.510778 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:46.510742    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:08:46.511026 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:46.510813    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:08:46.511026 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:46.510841    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:08:46.883202 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:46.883112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z" event={"ID":"2d099fda-4a87-45cb-a88f-a1d546cf1e6e","Type":"ContainerStarted","Data":"386f9a5928689f9017b11e3aa9f612f113128db4e459824552c2d106cecb5cda"}
Jul 28 20:08:46.903528 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:46.903479    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z" podStartSLOduration=1.8301344529999999 podStartE2EDuration="33.903461631s" totalImagesPullingTime="32.073327178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:46.902720865 +0000 UTC m=+576.757031935" watchObservedRunningTime="2026-07-28 20:08:46.903461631 +0000 UTC m=+576.757772698"
Jul 28 20:08:47.290907 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:47.290870    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:47.295609 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:47.295583    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:47.887900 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:47.887872    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:47.888708 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:47.888687    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:08:47.992376 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:47.992349    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"]
Jul 28 20:08:47.992923 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:47.992898    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:47.995750 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:47.995734    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:08:47.998160 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:47.998141    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dspjd\""
Jul 28 20:08:47.998260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:47.998174    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 20:08:48.002891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.002872    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"]
Jul 28 20:08:48.085988 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.085955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b59a0803-eb48-4537-be1a-f41a0aaed0b4-config-file\") pod \"limitador-limitador-64c94fd64f-rmpgd\" (UID: \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:08:48.085988 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.085989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxht5\" (UniqueName: \"kubernetes.io/projected/b59a0803-eb48-4537-be1a-f41a0aaed0b4-kube-api-access-rxht5\") pod \"limitador-limitador-64c94fd64f-rmpgd\" (UID: \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:08:48.089478 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.089447    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"]
Jul 28 20:08:48.098154 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.098130    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:48.153673 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.153611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:48.186695 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.186666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b59a0803-eb48-4537-be1a-f41a0aaed0b4-config-file\") pod \"limitador-limitador-64c94fd64f-rmpgd\" (UID: \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:08:48.186822 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.186707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxht5\" (UniqueName: \"kubernetes.io/projected/b59a0803-eb48-4537-be1a-f41a0aaed0b4-kube-api-access-rxht5\") pod \"limitador-limitador-64c94fd64f-rmpgd\" (UID: \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:08:48.187421 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.187400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b59a0803-eb48-4537-be1a-f41a0aaed0b4-config-file\") pod \"limitador-limitador-64c94fd64f-rmpgd\" (UID: \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:08:48.196427 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.196381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxht5\" (UniqueName: \"kubernetes.io/projected/b59a0803-eb48-4537-be1a-f41a0aaed0b4-kube-api-access-rxht5\") pod \"limitador-limitador-64c94fd64f-rmpgd\" (UID: \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:08:48.306671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.306631    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:08:48.440725 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.440700    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"]
Jul 28 20:08:48.442676 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:08:48.442646    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59a0803_eb48_4537_be1a_f41a0aaed0b4.slice/crio-c4581d8899ee9b47b5f4e5a8f7eba5f89d9d802f9565453b87fb8a9266c2da4a WatchSource:0}: Error finding container c4581d8899ee9b47b5f4e5a8f7eba5f89d9d802f9565453b87fb8a9266c2da4a: Status 404 returned error can't find the container with id c4581d8899ee9b47b5f4e5a8f7eba5f89d9d802f9565453b87fb8a9266c2da4a
Jul 28 20:08:48.778578 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.778550    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-w9cwn"]
Jul 28 20:08:48.782977 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.782958    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-w9cwn"
Jul 28 20:08:48.785932 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.785911    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hmxqt\""
Jul 28 20:08:48.789796 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.789767    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-w9cwn"]
Jul 28 20:08:48.818926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.818903    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:48.892194 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.892164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbcld\" (UniqueName: \"kubernetes.io/projected/c3d76908-31a5-4f87-b96f-440974666c52-kube-api-access-kbcld\") pod \"authorino-58cd955675-w9cwn\" (UID: \"c3d76908-31a5-4f87-b96f-440974666c52\") " pod="kuadrant-system/authorino-58cd955675-w9cwn"
Jul 28 20:08:48.893118 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.893082    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd" event={"ID":"b59a0803-eb48-4537-be1a-f41a0aaed0b4","Type":"ContainerStarted","Data":"c4581d8899ee9b47b5f4e5a8f7eba5f89d9d802f9565453b87fb8a9266c2da4a"}
Jul 28 20:08:48.907807 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.907785    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:48.992701 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:48.992666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbcld\" (UniqueName: \"kubernetes.io/projected/c3d76908-31a5-4f87-b96f-440974666c52-kube-api-access-kbcld\") pod \"authorino-58cd955675-w9cwn\" (UID: \"c3d76908-31a5-4f87-b96f-440974666c52\") " pod="kuadrant-system/authorino-58cd955675-w9cwn"
Jul 28 20:08:49.000906 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:49.000871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbcld\" (UniqueName: \"kubernetes.io/projected/c3d76908-31a5-4f87-b96f-440974666c52-kube-api-access-kbcld\") pod \"authorino-58cd955675-w9cwn\" (UID: \"c3d76908-31a5-4f87-b96f-440974666c52\") " pod="kuadrant-system/authorino-58cd955675-w9cwn"
Jul 28 20:08:49.093324 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:49.093230    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-w9cwn"
Jul 28 20:08:49.263071 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:49.262488    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-w9cwn"]
Jul 28 20:08:49.266504 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:08:49.266471    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3d76908_31a5_4f87_b96f_440974666c52.slice/crio-60f42e018cbdef6d93625b654549fb517d1e5d0b399a74417269a8b69dd5951c WatchSource:0}: Error finding container 60f42e018cbdef6d93625b654549fb517d1e5d0b399a74417269a8b69dd5951c: Status 404 returned error can't find the container with id 60f42e018cbdef6d93625b654549fb517d1e5d0b399a74417269a8b69dd5951c
Jul 28 20:08:49.898842 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:49.898804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-w9cwn" event={"ID":"c3d76908-31a5-4f87-b96f-440974666c52","Type":"ContainerStarted","Data":"60f42e018cbdef6d93625b654549fb517d1e5d0b399a74417269a8b69dd5951c"}
Jul 28 20:08:52.510755 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:52.510715    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-w9cwn"]
Jul 28 20:08:53.923508 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:53.923473    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-w9cwn" event={"ID":"c3d76908-31a5-4f87-b96f-440974666c52","Type":"ContainerStarted","Data":"291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320"}
Jul 28 20:08:53.923975 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:53.923511    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-w9cwn" podUID="c3d76908-31a5-4f87-b96f-440974666c52" containerName="authorino" containerID="cri-o://291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320" gracePeriod=30
Jul 28 20:08:53.925265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:53.925242    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd" event={"ID":"b59a0803-eb48-4537-be1a-f41a0aaed0b4","Type":"ContainerStarted","Data":"749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01"}
Jul 28 20:08:53.925391 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:53.925314    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:08:53.938796 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:53.938757    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-w9cwn" podStartSLOduration=2.06923733 podStartE2EDuration="5.938744034s" totalImagesPullingTime="3.869506704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:53.937392475 +0000 UTC m=+583.791703746" watchObservedRunningTime="2026-07-28 20:08:53.938744034 +0000 UTC m=+583.793055100"
Jul 28 20:08:53.952735 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:53.952703    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd" podStartSLOduration=2.259944989 podStartE2EDuration="6.952687624s" totalImagesPullingTime="4.692742635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:53.952338449 +0000 UTC m=+583.806649515" watchObservedRunningTime="2026-07-28 20:08:53.952687624 +0000 UTC m=+583.806998690"
Jul 28 20:08:54.168218 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.168195    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-w9cwn"
Jul 28 20:08:54.233429 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.233402    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbcld\" (UniqueName: \"kubernetes.io/projected/c3d76908-31a5-4f87-b96f-440974666c52-kube-api-access-kbcld\") pod \"c3d76908-31a5-4f87-b96f-440974666c52\" (UID: \"c3d76908-31a5-4f87-b96f-440974666c52\") "
Jul 28 20:08:54.235337 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.235312    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d76908-31a5-4f87-b96f-440974666c52-kube-api-access-kbcld" (OuterVolumeSpecName: "kube-api-access-kbcld") pod "c3d76908-31a5-4f87-b96f-440974666c52" (UID: "c3d76908-31a5-4f87-b96f-440974666c52"). InnerVolumeSpecName "kube-api-access-kbcld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:08:54.333920 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.333888    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbcld\" (UniqueName: \"kubernetes.io/projected/c3d76908-31a5-4f87-b96f-440974666c52-kube-api-access-kbcld\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:08:54.930598 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.930514    2582 generic.go:356] "Generic (PLEG): container finished" podID="c3d76908-31a5-4f87-b96f-440974666c52" containerID="291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320" exitCode=0
Jul 28 20:08:54.930598 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.930557    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-w9cwn"
Jul 28 20:08:54.931089 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.930599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-w9cwn" event={"ID":"c3d76908-31a5-4f87-b96f-440974666c52","Type":"ContainerDied","Data":"291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320"}
Jul 28 20:08:54.931089 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.930650    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-w9cwn" event={"ID":"c3d76908-31a5-4f87-b96f-440974666c52","Type":"ContainerDied","Data":"60f42e018cbdef6d93625b654549fb517d1e5d0b399a74417269a8b69dd5951c"}
Jul 28 20:08:54.931089 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.930676    2582 scope.go:117] "RemoveContainer" containerID="291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320"
Jul 28 20:08:54.942027 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.942006    2582 scope.go:117] "RemoveContainer" containerID="291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320"
Jul 28 20:08:54.942264 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:08:54.942246    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320\": container with ID starting with 291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320 not found: ID does not exist" containerID="291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320"
Jul 28 20:08:54.942316 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.942272    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320"} err="failed to get container status \"291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320\": rpc error: code = NotFound desc = could not find container \"291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320\": container with ID starting with 291e72f9efdf28f1b424dd7d6d62cf36e1c338aa0330e23f05355b09b60c3320 not found: ID does not exist"
Jul 28 20:08:54.949325 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.949284    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-w9cwn"]
Jul 28 20:08:54.953864 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:54.953842    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-w9cwn"]
Jul 28 20:08:56.582301 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:08:56.582266    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3d76908-31a5-4f87-b96f-440974666c52" path="/var/lib/kubelet/pods/c3d76908-31a5-4f87-b96f-440974666c52/volumes"
Jul 28 20:09:03.027202 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.027171    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"]
Jul 28 20:09:03.027683 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.027372    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd" podUID="b59a0803-eb48-4537-be1a-f41a0aaed0b4" containerName="limitador" containerID="cri-o://749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01" gracePeriod=30
Jul 28 20:09:03.028040 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.027977    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:09:03.576626 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.576602    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:09:03.710320 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.710228    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxht5\" (UniqueName: \"kubernetes.io/projected/b59a0803-eb48-4537-be1a-f41a0aaed0b4-kube-api-access-rxht5\") pod \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\" (UID: \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\") "
Jul 28 20:09:03.710465 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.710334    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b59a0803-eb48-4537-be1a-f41a0aaed0b4-config-file\") pod \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\" (UID: \"b59a0803-eb48-4537-be1a-f41a0aaed0b4\") "
Jul 28 20:09:03.710689 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.710670    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59a0803-eb48-4537-be1a-f41a0aaed0b4-config-file" (OuterVolumeSpecName: "config-file") pod "b59a0803-eb48-4537-be1a-f41a0aaed0b4" (UID: "b59a0803-eb48-4537-be1a-f41a0aaed0b4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:09:03.712278 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.712245    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59a0803-eb48-4537-be1a-f41a0aaed0b4-kube-api-access-rxht5" (OuterVolumeSpecName: "kube-api-access-rxht5") pod "b59a0803-eb48-4537-be1a-f41a0aaed0b4" (UID: "b59a0803-eb48-4537-be1a-f41a0aaed0b4"). InnerVolumeSpecName "kube-api-access-rxht5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:09:03.811330 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.811285    2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b59a0803-eb48-4537-be1a-f41a0aaed0b4-config-file\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:09:03.811330 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.811325    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxht5\" (UniqueName: \"kubernetes.io/projected/b59a0803-eb48-4537-be1a-f41a0aaed0b4-kube-api-access-rxht5\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:09:03.972514 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.972437    2582 generic.go:356] "Generic (PLEG): container finished" podID="b59a0803-eb48-4537-be1a-f41a0aaed0b4" containerID="749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01" exitCode=0
Jul 28 20:09:03.972514 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.972481    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"
Jul 28 20:09:03.972680 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.972528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd" event={"ID":"b59a0803-eb48-4537-be1a-f41a0aaed0b4","Type":"ContainerDied","Data":"749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01"}
Jul 28 20:09:03.972680 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.972582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmpgd" event={"ID":"b59a0803-eb48-4537-be1a-f41a0aaed0b4","Type":"ContainerDied","Data":"c4581d8899ee9b47b5f4e5a8f7eba5f89d9d802f9565453b87fb8a9266c2da4a"}
Jul 28 20:09:03.972680 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.972609    2582 scope.go:117] "RemoveContainer" containerID="749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01"
Jul 28 20:09:03.981142 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.981125    2582 scope.go:117] "RemoveContainer" containerID="749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01"
Jul 28 20:09:03.981433 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:09:03.981413    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01\": container with ID starting with 749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01 not found: ID does not exist" containerID="749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01"
Jul 28 20:09:03.981499 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.981443    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01"} err="failed to get container status \"749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01\": rpc error: code = NotFound desc = could not find container \"749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01\": container with ID starting with 749830d2187b10aa0b59f0e742c20f060f5d771c5ac0ff0e2aca72e6536b8c01 not found: ID does not exist"
Jul 28 20:09:03.993050 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:03.993029    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"]
Jul 28 20:09:04.000024 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:04.000000    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmpgd"]
Jul 28 20:09:04.582010 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:04.581979    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b59a0803-eb48-4537-be1a-f41a0aaed0b4" path="/var/lib/kubelet/pods/b59a0803-eb48-4537-be1a-f41a0aaed0b4/volumes"
Jul 28 20:09:09.132348 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.132309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.232643 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.232612    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.573477 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.573438    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"]
Jul 28 20:09:09.573754 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.573740    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59a0803-eb48-4537-be1a-f41a0aaed0b4" containerName="limitador"
Jul 28 20:09:09.573806 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.573757    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59a0803-eb48-4537-be1a-f41a0aaed0b4" containerName="limitador"
Jul 28 20:09:09.573806 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.573778    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d76908-31a5-4f87-b96f-440974666c52" containerName="authorino"
Jul 28 20:09:09.573806 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.573784    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d76908-31a5-4f87-b96f-440974666c52" containerName="authorino"
Jul 28 20:09:09.573894 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.573842    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b59a0803-eb48-4537-be1a-f41a0aaed0b4" containerName="limitador"
Jul 28 20:09:09.573894 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.573852    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d76908-31a5-4f87-b96f-440974666c52" containerName="authorino"
Jul 28 20:09:09.576911 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.576882    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:09.579610 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.579585    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:09:09.579610 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.579598    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jvd2r\""
Jul 28 20:09:09.579804 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.579665    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 28 20:09:09.580740 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.580722    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:09:09.584087 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.584045    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"]
Jul 28 20:09:09.613437 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.613413    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.661028 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.661002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n99hn\" (UniqueName: \"kubernetes.io/projected/e8d74642-814c-4ac1-8b5d-a228d7736d79-kube-api-access-n99hn\") pod \"postgres-69fb4594cf-wxz7t\" (UID: \"e8d74642-814c-4ac1-8b5d-a228d7736d79\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:09.661136 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.661044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e8d74642-814c-4ac1-8b5d-a228d7736d79-data\") pod \"postgres-69fb4594cf-wxz7t\" (UID: \"e8d74642-814c-4ac1-8b5d-a228d7736d79\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:09.762031 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.761998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n99hn\" (UniqueName: \"kubernetes.io/projected/e8d74642-814c-4ac1-8b5d-a228d7736d79-kube-api-access-n99hn\") pod \"postgres-69fb4594cf-wxz7t\" (UID: \"e8d74642-814c-4ac1-8b5d-a228d7736d79\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:09.762192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.762041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e8d74642-814c-4ac1-8b5d-a228d7736d79-data\") pod \"postgres-69fb4594cf-wxz7t\" (UID: \"e8d74642-814c-4ac1-8b5d-a228d7736d79\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:09.762403 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.762386    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e8d74642-814c-4ac1-8b5d-a228d7736d79-data\") pod \"postgres-69fb4594cf-wxz7t\" (UID: \"e8d74642-814c-4ac1-8b5d-a228d7736d79\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:09.769818 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.769800    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n99hn\" (UniqueName: \"kubernetes.io/projected/e8d74642-814c-4ac1-8b5d-a228d7736d79-kube-api-access-n99hn\") pod \"postgres-69fb4594cf-wxz7t\" (UID: \"e8d74642-814c-4ac1-8b5d-a228d7736d79\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:09.889211 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:09.889145    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:10.016789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:10.016768    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"]
Jul 28 20:09:10.018796 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:09:10.018768    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d74642_814c_4ac1_8b5d_a228d7736d79.slice/crio-abc9328953bb5b9c39c29e4909ffd946f1ca7f79d233e7766dc63a173a85bbfe WatchSource:0}: Error finding container abc9328953bb5b9c39c29e4909ffd946f1ca7f79d233e7766dc63a173a85bbfe: Status 404 returned error can't find the container with id abc9328953bb5b9c39c29e4909ffd946f1ca7f79d233e7766dc63a173a85bbfe
Jul 28 20:09:11.002615 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:11.002578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t" event={"ID":"e8d74642-814c-4ac1-8b5d-a228d7736d79","Type":"ContainerStarted","Data":"abc9328953bb5b9c39c29e4909ffd946f1ca7f79d233e7766dc63a173a85bbfe"}
Jul 28 20:09:14.705142 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:14.705107    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:15.050700 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:15.050677    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 28 20:09:16.027022 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:16.026990    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t" event={"ID":"e8d74642-814c-4ac1-8b5d-a228d7736d79","Type":"ContainerStarted","Data":"6766f0b05439163eb00f0ddc1bc6c0525e70a2b3a4d865f444d73f006014d270"}
Jul 28 20:09:16.027405 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:16.027075    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:16.042525 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:16.042482    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t" podStartSLOduration=2.014645483 podStartE2EDuration="7.042467714s" totalImagesPullingTime="5.027822231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:16.041632165 +0000 UTC m=+605.895943232" watchObservedRunningTime="2026-07-28 20:09:16.042467714 +0000 UTC m=+605.896778794"
Jul 28 20:09:22.058972 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.058943    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxz7t"
Jul 28 20:09:22.366003 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.365923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:22.595507 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.595477    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pgq62"]
Jul 28 20:09:22.602173 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.602152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pgq62"
Jul 28 20:09:22.604794 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.604769    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hmxqt\""
Jul 28 20:09:22.605257 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.605231    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pgq62"]
Jul 28 20:09:22.664315 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.664222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:22.672385 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.672362    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jvv\" (UniqueName: \"kubernetes.io/projected/262ce2e1-4311-47a3-b2d5-51d624ac599f-kube-api-access-97jvv\") pod \"authorino-5d4b67c446-pgq62\" (UID: \"262ce2e1-4311-47a3-b2d5-51d624ac599f\") " pod="kuadrant-system/authorino-5d4b67c446-pgq62"
Jul 28 20:09:22.773267 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.773232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jvv\" (UniqueName: \"kubernetes.io/projected/262ce2e1-4311-47a3-b2d5-51d624ac599f-kube-api-access-97jvv\") pod \"authorino-5d4b67c446-pgq62\" (UID: \"262ce2e1-4311-47a3-b2d5-51d624ac599f\") " pod="kuadrant-system/authorino-5d4b67c446-pgq62"
Jul 28 20:09:22.783073 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.783036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jvv\" (UniqueName: \"kubernetes.io/projected/262ce2e1-4311-47a3-b2d5-51d624ac599f-kube-api-access-97jvv\") pod \"authorino-5d4b67c446-pgq62\" (UID: \"262ce2e1-4311-47a3-b2d5-51d624ac599f\") " pod="kuadrant-system/authorino-5d4b67c446-pgq62"
Jul 28 20:09:22.797183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.797154    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pgq62"]
Jul 28 20:09:22.797436 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.797421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pgq62"
Jul 28 20:09:22.823646 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.823620    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-5fvd7"]
Jul 28 20:09:22.829703 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.829680    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:22.832146 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.832127    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 20:09:22.835396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.835375    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-5fvd7"]
Jul 28 20:09:22.855773 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.855745    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-5fvd7"]
Jul 28 20:09:22.856161 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:09:22.856114    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ljm8x tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-ljm8x tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-5fvd7" podUID="c95f130e-b5aa-43e9-b5e7-aef180f2bd38"
Jul 28 20:09:22.886572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.886541    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55cbc584f7-h8vbj"]
Jul 28 20:09:22.893127 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.893102    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:09:22.898860 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.898793    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55cbc584f7-h8vbj"]
Jul 28 20:09:22.936667 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.936592    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pgq62"]
Jul 28 20:09:22.940274 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:09:22.940251    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod262ce2e1_4311_47a3_b2d5_51d624ac599f.slice/crio-081ab9222911270e119a445daafc420bdf948436413a108b2025a7b8c04fc54c WatchSource:0}: Error finding container 081ab9222911270e119a445daafc420bdf948436413a108b2025a7b8c04fc54c: Status 404 returned error can't find the container with id 081ab9222911270e119a445daafc420bdf948436413a108b2025a7b8c04fc54c
Jul 28 20:09:22.974827 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.974801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28dgv\" (UniqueName: \"kubernetes.io/projected/8661cb26-3520-4297-8d93-66a4e1395d57-kube-api-access-28dgv\") pod \"authorino-55cbc584f7-h8vbj\" (UID: \"8661cb26-3520-4297-8d93-66a4e1395d57\") " pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:09:22.974914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.974859    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljm8x\" (UniqueName: \"kubernetes.io/projected/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-kube-api-access-ljm8x\") pod \"authorino-76877994-5fvd7\" (UID: \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\") " pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:22.974914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.974880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8661cb26-3520-4297-8d93-66a4e1395d57-tls-cert\") pod \"authorino-55cbc584f7-h8vbj\" (UID: \"8661cb26-3520-4297-8d93-66a4e1395d57\") " pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:09:22.974914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:22.974912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-tls-cert\") pod \"authorino-76877994-5fvd7\" (UID: \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\") " pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:23.058478 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.058440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pgq62" event={"ID":"262ce2e1-4311-47a3-b2d5-51d624ac599f","Type":"ContainerStarted","Data":"081ab9222911270e119a445daafc420bdf948436413a108b2025a7b8c04fc54c"}
Jul 28 20:09:23.058478 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.058474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:23.069554 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.069525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:23.075275 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.075248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28dgv\" (UniqueName: \"kubernetes.io/projected/8661cb26-3520-4297-8d93-66a4e1395d57-kube-api-access-28dgv\") pod \"authorino-55cbc584f7-h8vbj\" (UID: \"8661cb26-3520-4297-8d93-66a4e1395d57\") " pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:09:23.075467 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.075335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljm8x\" (UniqueName: \"kubernetes.io/projected/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-kube-api-access-ljm8x\") pod \"authorino-76877994-5fvd7\" (UID: \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\") " pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:23.075467 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.075368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8661cb26-3520-4297-8d93-66a4e1395d57-tls-cert\") pod \"authorino-55cbc584f7-h8vbj\" (UID: \"8661cb26-3520-4297-8d93-66a4e1395d57\") " pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:09:23.075467 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.075392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-tls-cert\") pod \"authorino-76877994-5fvd7\" (UID: \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\") " pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:23.077811 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.077790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8661cb26-3520-4297-8d93-66a4e1395d57-tls-cert\") pod \"authorino-55cbc584f7-h8vbj\" (UID: \"8661cb26-3520-4297-8d93-66a4e1395d57\") " pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:09:23.077926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.077870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-tls-cert\") pod \"authorino-76877994-5fvd7\" (UID: \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\") " pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:23.082733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.082705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljm8x\" (UniqueName: \"kubernetes.io/projected/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-kube-api-access-ljm8x\") pod \"authorino-76877994-5fvd7\" (UID: \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\") " pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:23.083002 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.082974    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28dgv\" (UniqueName: \"kubernetes.io/projected/8661cb26-3520-4297-8d93-66a4e1395d57-kube-api-access-28dgv\") pod \"authorino-55cbc584f7-h8vbj\" (UID: \"8661cb26-3520-4297-8d93-66a4e1395d57\") " pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:09:23.176310 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.176259    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljm8x\" (UniqueName: \"kubernetes.io/projected/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-kube-api-access-ljm8x\") pod \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\" (UID: \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\") "
Jul 28 20:09:23.176472 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.176334    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-tls-cert\") pod \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\" (UID: \"c95f130e-b5aa-43e9-b5e7-aef180f2bd38\") "
Jul 28 20:09:23.178237 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.178209    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c95f130e-b5aa-43e9-b5e7-aef180f2bd38" (UID: "c95f130e-b5aa-43e9-b5e7-aef180f2bd38"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:09:23.178329 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.178251    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-kube-api-access-ljm8x" (OuterVolumeSpecName: "kube-api-access-ljm8x") pod "c95f130e-b5aa-43e9-b5e7-aef180f2bd38" (UID: "c95f130e-b5aa-43e9-b5e7-aef180f2bd38"). InnerVolumeSpecName "kube-api-access-ljm8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:09:23.204471 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.204406    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:09:23.277737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.277675    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljm8x\" (UniqueName: \"kubernetes.io/projected/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-kube-api-access-ljm8x\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:09:23.277737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.277710    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c95f130e-b5aa-43e9-b5e7-aef180f2bd38-tls-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:09:23.354916 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:23.354840    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55cbc584f7-h8vbj"]
Jul 28 20:09:23.356958 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:09:23.356932    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8661cb26_3520_4297_8d93_66a4e1395d57.slice/crio-263fcc6733da6bb06f1eb2702573dc6ddeb9155301424fe0e19f9a90861c2818 WatchSource:0}: Error finding container 263fcc6733da6bb06f1eb2702573dc6ddeb9155301424fe0e19f9a90861c2818: Status 404 returned error can't find the container with id 263fcc6733da6bb06f1eb2702573dc6ddeb9155301424fe0e19f9a90861c2818
Jul 28 20:09:24.064005 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.063919    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55cbc584f7-h8vbj" event={"ID":"8661cb26-3520-4297-8d93-66a4e1395d57","Type":"ContainerStarted","Data":"263fcc6733da6bb06f1eb2702573dc6ddeb9155301424fe0e19f9a90861c2818"}
Jul 28 20:09:24.065237 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.065204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pgq62" event={"ID":"262ce2e1-4311-47a3-b2d5-51d624ac599f","Type":"ContainerStarted","Data":"d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f"}
Jul 28 20:09:24.065237 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.065223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-5fvd7"
Jul 28 20:09:24.065237 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.065232    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-pgq62" podUID="262ce2e1-4311-47a3-b2d5-51d624ac599f" containerName="authorino" containerID="cri-o://d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f" gracePeriod=30
Jul 28 20:09:24.080872 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.080807    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-pgq62" podStartSLOduration=1.503322756 podStartE2EDuration="2.080792888s" totalImagesPullingTime="577.470132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:24.079172333 +0000 UTC m=+613.933483400" watchObservedRunningTime="2026-07-28 20:09:24.080792888 +0000 UTC m=+613.935103945"
Jul 28 20:09:24.101801 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.101775    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-5fvd7"]
Jul 28 20:09:24.105595 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.105571    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-5fvd7"]
Jul 28 20:09:24.358592 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.358567    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pgq62"
Jul 28 20:09:24.488609 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.488577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97jvv\" (UniqueName: \"kubernetes.io/projected/262ce2e1-4311-47a3-b2d5-51d624ac599f-kube-api-access-97jvv\") pod \"262ce2e1-4311-47a3-b2d5-51d624ac599f\" (UID: \"262ce2e1-4311-47a3-b2d5-51d624ac599f\") "
Jul 28 20:09:24.490800 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.490777    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262ce2e1-4311-47a3-b2d5-51d624ac599f-kube-api-access-97jvv" (OuterVolumeSpecName: "kube-api-access-97jvv") pod "262ce2e1-4311-47a3-b2d5-51d624ac599f" (UID: "262ce2e1-4311-47a3-b2d5-51d624ac599f"). InnerVolumeSpecName "kube-api-access-97jvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:09:24.582880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.582808    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c95f130e-b5aa-43e9-b5e7-aef180f2bd38" path="/var/lib/kubelet/pods/c95f130e-b5aa-43e9-b5e7-aef180f2bd38/volumes"
Jul 28 20:09:24.590048 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:24.590023    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jvv\" (UniqueName: \"kubernetes.io/projected/262ce2e1-4311-47a3-b2d5-51d624ac599f-kube-api-access-97jvv\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:09:25.071495 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.071458    2582 generic.go:356] "Generic (PLEG): container finished" podID="262ce2e1-4311-47a3-b2d5-51d624ac599f" containerID="d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f" exitCode=0
Jul 28 20:09:25.071647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.071501    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pgq62"
Jul 28 20:09:25.071647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.071536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pgq62" event={"ID":"262ce2e1-4311-47a3-b2d5-51d624ac599f","Type":"ContainerDied","Data":"d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f"}
Jul 28 20:09:25.071647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.071584    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pgq62" event={"ID":"262ce2e1-4311-47a3-b2d5-51d624ac599f","Type":"ContainerDied","Data":"081ab9222911270e119a445daafc420bdf948436413a108b2025a7b8c04fc54c"}
Jul 28 20:09:25.071647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.071609    2582 scope.go:117] "RemoveContainer" containerID="d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f"
Jul 28 20:09:25.073284 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.073259    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55cbc584f7-h8vbj" event={"ID":"8661cb26-3520-4297-8d93-66a4e1395d57","Type":"ContainerStarted","Data":"1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28"}
Jul 28 20:09:25.079551 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.079536    2582 scope.go:117] "RemoveContainer" containerID="d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f"
Jul 28 20:09:25.079792 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:09:25.079773    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f\": container with ID starting with d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f not found: ID does not exist" containerID="d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f"
Jul 28 20:09:25.079835 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.079800    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f"} err="failed to get container status \"d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f\": rpc error: code = NotFound desc = could not find container \"d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f\": container with ID starting with d06508c06f3c82f7e97a4ddcbcee1cbf5d0ff56d533507aad041e5c3231e855f not found: ID does not exist"
Jul 28 20:09:25.091645 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.091604    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55cbc584f7-h8vbj" podStartSLOduration=2.33611515 podStartE2EDuration="3.091591714s" totalImagesPullingTime="755.476564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:25.089806888 +0000 UTC m=+614.944117954" watchObservedRunningTime="2026-07-28 20:09:25.091591714 +0000 UTC m=+614.945902780"
Jul 28 20:09:25.104660 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.104639    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pgq62"]
Jul 28 20:09:25.111381 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:25.111352    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pgq62"]
Jul 28 20:09:26.582115 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:26.582082    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="262ce2e1-4311-47a3-b2d5-51d624ac599f" path="/var/lib/kubelet/pods/262ce2e1-4311-47a3-b2d5-51d624ac599f/volumes"
Jul 28 20:09:29.567789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.567756    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:29.582913 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.582886    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74c48f9d99-jhlk9"]
Jul 28 20:09:29.583255 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.583238    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="262ce2e1-4311-47a3-b2d5-51d624ac599f" containerName="authorino"
Jul 28 20:09:29.583354 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.583257    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="262ce2e1-4311-47a3-b2d5-51d624ac599f" containerName="authorino"
Jul 28 20:09:29.583422 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.583410    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="262ce2e1-4311-47a3-b2d5-51d624ac599f" containerName="authorino"
Jul 28 20:09:29.587064 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.587044    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:29.589920 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.589898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vc59v\""
Jul 28 20:09:29.590016 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.589921    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:09:29.590016 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.589928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:09:29.599175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.599156    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74c48f9d99-jhlk9"]
Jul 28 20:09:29.664087 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.664055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:29.721969 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.721939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:29.731106 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.731079    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f05e297-875b-4110-8361-aaa2f43f1bdb-webhook-cert\") pod \"maas-controller-74c48f9d99-jhlk9\" (UID: \"0f05e297-875b-4110-8361-aaa2f43f1bdb\") " pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:29.731240 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.731215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttlb7\" (UniqueName: \"kubernetes.io/projected/0f05e297-875b-4110-8361-aaa2f43f1bdb-kube-api-access-ttlb7\") pod \"maas-controller-74c48f9d99-jhlk9\" (UID: \"0f05e297-875b-4110-8361-aaa2f43f1bdb\") " pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:29.762894 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.762862    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58546478d9-tzj6c"]
Jul 28 20:09:29.765849 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.765833    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:29.774911 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.774882    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58546478d9-tzj6c"]
Jul 28 20:09:29.792100 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.792056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:29.831866 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.831810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f05e297-875b-4110-8361-aaa2f43f1bdb-webhook-cert\") pod \"maas-controller-74c48f9d99-jhlk9\" (UID: \"0f05e297-875b-4110-8361-aaa2f43f1bdb\") " pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:29.831866 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.831862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttlb7\" (UniqueName: \"kubernetes.io/projected/0f05e297-875b-4110-8361-aaa2f43f1bdb-kube-api-access-ttlb7\") pod \"maas-controller-74c48f9d99-jhlk9\" (UID: \"0f05e297-875b-4110-8361-aaa2f43f1bdb\") " pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:29.834237 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.834216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f05e297-875b-4110-8361-aaa2f43f1bdb-webhook-cert\") pod \"maas-controller-74c48f9d99-jhlk9\" (UID: \"0f05e297-875b-4110-8361-aaa2f43f1bdb\") " pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:29.840974 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.840951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttlb7\" (UniqueName: \"kubernetes.io/projected/0f05e297-875b-4110-8361-aaa2f43f1bdb-kube-api-access-ttlb7\") pod \"maas-controller-74c48f9d99-jhlk9\" (UID: \"0f05e297-875b-4110-8361-aaa2f43f1bdb\") " pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:29.897928 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.897903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:29.932903 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.932870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbzf8\" (UniqueName: \"kubernetes.io/projected/2846dc92-55a8-4be0-8906-6233c33d212c-kube-api-access-xbzf8\") pod \"maas-controller-58546478d9-tzj6c\" (UID: \"2846dc92-55a8-4be0-8906-6233c33d212c\") " pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:29.933063 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:29.932926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2846dc92-55a8-4be0-8906-6233c33d212c-webhook-cert\") pod \"maas-controller-58546478d9-tzj6c\" (UID: \"2846dc92-55a8-4be0-8906-6233c33d212c\") " pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:30.023718 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:30.023690    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74c48f9d99-jhlk9"]
Jul 28 20:09:30.025587 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:09:30.025562    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f05e297_875b_4110_8361_aaa2f43f1bdb.slice/crio-368ccc950df8f8508dcc61ef02bdde2500e318e05b13705d0d6647927dae18f3 WatchSource:0}: Error finding container 368ccc950df8f8508dcc61ef02bdde2500e318e05b13705d0d6647927dae18f3: Status 404 returned error can't find the container with id 368ccc950df8f8508dcc61ef02bdde2500e318e05b13705d0d6647927dae18f3
Jul 28 20:09:30.033941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:30.033918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbzf8\" (UniqueName: \"kubernetes.io/projected/2846dc92-55a8-4be0-8906-6233c33d212c-kube-api-access-xbzf8\") pod \"maas-controller-58546478d9-tzj6c\" (UID: \"2846dc92-55a8-4be0-8906-6233c33d212c\") " pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:30.034060 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:30.033968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2846dc92-55a8-4be0-8906-6233c33d212c-webhook-cert\") pod \"maas-controller-58546478d9-tzj6c\" (UID: \"2846dc92-55a8-4be0-8906-6233c33d212c\") " pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:30.036194 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:30.036174    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2846dc92-55a8-4be0-8906-6233c33d212c-webhook-cert\") pod \"maas-controller-58546478d9-tzj6c\" (UID: \"2846dc92-55a8-4be0-8906-6233c33d212c\") " pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:30.041683 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:30.041662    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbzf8\" (UniqueName: \"kubernetes.io/projected/2846dc92-55a8-4be0-8906-6233c33d212c-kube-api-access-xbzf8\") pod \"maas-controller-58546478d9-tzj6c\" (UID: \"2846dc92-55a8-4be0-8906-6233c33d212c\") " pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:30.077185 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:30.077149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:30.097073 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:30.097041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74c48f9d99-jhlk9" event={"ID":"0f05e297-875b-4110-8361-aaa2f43f1bdb","Type":"ContainerStarted","Data":"368ccc950df8f8508dcc61ef02bdde2500e318e05b13705d0d6647927dae18f3"}
Jul 28 20:09:30.206581 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:30.206557    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58546478d9-tzj6c"]
Jul 28 20:09:30.208797 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:09:30.208770    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2846dc92_55a8_4be0_8906_6233c33d212c.slice/crio-1351920a13b0215e7a0f6d518669ef30ae59120ec4af45c9299ec034c0b27e50 WatchSource:0}: Error finding container 1351920a13b0215e7a0f6d518669ef30ae59120ec4af45c9299ec034c0b27e50: Status 404 returned error can't find the container with id 1351920a13b0215e7a0f6d518669ef30ae59120ec4af45c9299ec034c0b27e50
Jul 28 20:09:31.103365 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:31.103327    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-tzj6c" event={"ID":"2846dc92-55a8-4be0-8906-6233c33d212c","Type":"ContainerStarted","Data":"1351920a13b0215e7a0f6d518669ef30ae59120ec4af45c9299ec034c0b27e50"}
Jul 28 20:09:33.017052 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:33.017010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:33.115366 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:33.115255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74c48f9d99-jhlk9" event={"ID":"0f05e297-875b-4110-8361-aaa2f43f1bdb","Type":"ContainerStarted","Data":"c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca"}
Jul 28 20:09:33.115366 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:33.115354    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:33.116833 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:33.116807    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-tzj6c" event={"ID":"2846dc92-55a8-4be0-8906-6233c33d212c","Type":"ContainerStarted","Data":"9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4"}
Jul 28 20:09:33.116976 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:33.116830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:33.116976 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:33.116938    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:33.141779 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:33.141734    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74c48f9d99-jhlk9" podStartSLOduration=1.309800783 podStartE2EDuration="4.141722924s" totalImagesPullingTime="2.831922141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:33.140005422 +0000 UTC m=+622.994316491" watchObservedRunningTime="2026-07-28 20:09:33.141722924 +0000 UTC m=+622.996033990"
Jul 28 20:09:33.163538 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:33.163491    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58546478d9-tzj6c" podStartSLOduration=1.5079162720000001 podStartE2EDuration="4.163476054s" totalImagesPullingTime="2.655559782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:33.162396147 +0000 UTC m=+623.016707224" watchObservedRunningTime="2026-07-28 20:09:33.163476054 +0000 UTC m=+623.017787119"
Jul 28 20:09:44.123702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.123673    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:09:44.126156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.123742    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:44.187516 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.187485    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74c48f9d99-jhlk9"]
Jul 28 20:09:44.187722 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.187697    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74c48f9d99-jhlk9" podUID="0f05e297-875b-4110-8361-aaa2f43f1bdb" containerName="manager" containerID="cri-o://c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca" gracePeriod=30
Jul 28 20:09:44.429221 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.429196    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:44.443822 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.443804    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttlb7\" (UniqueName: \"kubernetes.io/projected/0f05e297-875b-4110-8361-aaa2f43f1bdb-kube-api-access-ttlb7\") pod \"0f05e297-875b-4110-8361-aaa2f43f1bdb\" (UID: \"0f05e297-875b-4110-8361-aaa2f43f1bdb\") "
Jul 28 20:09:44.443923 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.443882    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f05e297-875b-4110-8361-aaa2f43f1bdb-webhook-cert\") pod \"0f05e297-875b-4110-8361-aaa2f43f1bdb\" (UID: \"0f05e297-875b-4110-8361-aaa2f43f1bdb\") "
Jul 28 20:09:44.446075 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.446048    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f05e297-875b-4110-8361-aaa2f43f1bdb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0f05e297-875b-4110-8361-aaa2f43f1bdb" (UID: "0f05e297-875b-4110-8361-aaa2f43f1bdb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:09:44.446191 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.446170    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f05e297-875b-4110-8361-aaa2f43f1bdb-kube-api-access-ttlb7" (OuterVolumeSpecName: "kube-api-access-ttlb7") pod "0f05e297-875b-4110-8361-aaa2f43f1bdb" (UID: "0f05e297-875b-4110-8361-aaa2f43f1bdb"). InnerVolumeSpecName "kube-api-access-ttlb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:09:44.544543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.544510    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttlb7\" (UniqueName: \"kubernetes.io/projected/0f05e297-875b-4110-8361-aaa2f43f1bdb-kube-api-access-ttlb7\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:09:44.544543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:44.544539    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f05e297-875b-4110-8361-aaa2f43f1bdb-webhook-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:09:45.175633 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:45.175594    2582 generic.go:356] "Generic (PLEG): container finished" podID="0f05e297-875b-4110-8361-aaa2f43f1bdb" containerID="c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca" exitCode=0
Jul 28 20:09:45.176048 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:45.175647    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74c48f9d99-jhlk9"
Jul 28 20:09:45.176048 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:45.175678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74c48f9d99-jhlk9" event={"ID":"0f05e297-875b-4110-8361-aaa2f43f1bdb","Type":"ContainerDied","Data":"c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca"}
Jul 28 20:09:45.176048 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:45.175736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74c48f9d99-jhlk9" event={"ID":"0f05e297-875b-4110-8361-aaa2f43f1bdb","Type":"ContainerDied","Data":"368ccc950df8f8508dcc61ef02bdde2500e318e05b13705d0d6647927dae18f3"}
Jul 28 20:09:45.176048 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:45.175758    2582 scope.go:117] "RemoveContainer" containerID="c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca"
Jul 28 20:09:45.184148 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:45.184132    2582 scope.go:117] "RemoveContainer" containerID="c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca"
Jul 28 20:09:45.184403 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:09:45.184385    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca\": container with ID starting with c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca not found: ID does not exist" containerID="c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca"
Jul 28 20:09:45.184458 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:45.184411    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca"} err="failed to get container status \"c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca\": rpc error: code = NotFound desc = could not find container \"c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca\": container with ID starting with c88a59dcf4a8aaa3209a73400507b3387b958e5ef345719c9edf0df6c4cfadca not found: ID does not exist"
Jul 28 20:09:45.195602 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:45.195579    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74c48f9d99-jhlk9"]
Jul 28 20:09:45.201458 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:45.201438    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74c48f9d99-jhlk9"]
Jul 28 20:09:46.582444 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:09:46.582413    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f05e297-875b-4110-8361-aaa2f43f1bdb" path="/var/lib/kubelet/pods/0f05e297-875b-4110-8361-aaa2f43f1bdb/volumes"
Jul 28 20:10:04.751256 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:04.751218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:04.844082 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:04.844022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:05.325466 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.325432    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:05.452732 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.452701    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"]
Jul 28 20:10:05.453156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.453137    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f05e297-875b-4110-8361-aaa2f43f1bdb" containerName="manager"
Jul 28 20:10:05.453270 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.453159    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f05e297-875b-4110-8361-aaa2f43f1bdb" containerName="manager"
Jul 28 20:10:05.453270 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.453232    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f05e297-875b-4110-8361-aaa2f43f1bdb" containerName="manager"
Jul 28 20:10:05.456031 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.456016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.459719 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.459700    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 28 20:10:05.459817 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.459702    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bqrm9\""
Jul 28 20:10:05.466536 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.466514    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"]
Jul 28 20:10:05.499478 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.499450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82f1086-427d-4a3c-ba1e-15aa78b38364-maas-api-tls\") pod \"maas-api-74bb79f4fd-7kzlx\" (UID: \"d82f1086-427d-4a3c-ba1e-15aa78b38364\") " pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.499577 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.499488    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82f1086-427d-4a3c-ba1e-15aa78b38364-openshift-service-ca\") pod \"maas-api-74bb79f4fd-7kzlx\" (UID: \"d82f1086-427d-4a3c-ba1e-15aa78b38364\") " pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.499577 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.499553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82gs6\" (UniqueName: \"kubernetes.io/projected/d82f1086-427d-4a3c-ba1e-15aa78b38364-kube-api-access-82gs6\") pod \"maas-api-74bb79f4fd-7kzlx\" (UID: \"d82f1086-427d-4a3c-ba1e-15aa78b38364\") " pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.500748 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.500722    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:05.600541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.600459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82gs6\" (UniqueName: \"kubernetes.io/projected/d82f1086-427d-4a3c-ba1e-15aa78b38364-kube-api-access-82gs6\") pod \"maas-api-74bb79f4fd-7kzlx\" (UID: \"d82f1086-427d-4a3c-ba1e-15aa78b38364\") " pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.600541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.600511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82f1086-427d-4a3c-ba1e-15aa78b38364-maas-api-tls\") pod \"maas-api-74bb79f4fd-7kzlx\" (UID: \"d82f1086-427d-4a3c-ba1e-15aa78b38364\") " pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.600541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.600538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82f1086-427d-4a3c-ba1e-15aa78b38364-openshift-service-ca\") pod \"maas-api-74bb79f4fd-7kzlx\" (UID: \"d82f1086-427d-4a3c-ba1e-15aa78b38364\") " pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.601129 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.601108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82f1086-427d-4a3c-ba1e-15aa78b38364-openshift-service-ca\") pod \"maas-api-74bb79f4fd-7kzlx\" (UID: \"d82f1086-427d-4a3c-ba1e-15aa78b38364\") " pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.602843 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.602823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82f1086-427d-4a3c-ba1e-15aa78b38364-maas-api-tls\") pod \"maas-api-74bb79f4fd-7kzlx\" (UID: \"d82f1086-427d-4a3c-ba1e-15aa78b38364\") " pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.608753 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.608732    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82gs6\" (UniqueName: \"kubernetes.io/projected/d82f1086-427d-4a3c-ba1e-15aa78b38364-kube-api-access-82gs6\") pod \"maas-api-74bb79f4fd-7kzlx\" (UID: \"d82f1086-427d-4a3c-ba1e-15aa78b38364\") " pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.766325 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.766271    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:05.835026 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.834995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:05.909623 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.909540    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"]
Jul 28 20:10:05.911877 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:10:05.911850    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82f1086_427d_4a3c_ba1e_15aa78b38364.slice/crio-69ccc49d875f9f2b0e30e6ef52590cfb0f68dca8f59a9374cab2e035a742a8f7 WatchSource:0}: Error finding container 69ccc49d875f9f2b0e30e6ef52590cfb0f68dca8f59a9374cab2e035a742a8f7: Status 404 returned error can't find the container with id 69ccc49d875f9f2b0e30e6ef52590cfb0f68dca8f59a9374cab2e035a742a8f7
Jul 28 20:10:05.913982 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.913959    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:05.985377 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.985232    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5gbgl"]
Jul 28 20:10:05.990501 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.990454    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5gbgl"]
Jul 28 20:10:05.990655 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.990571    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:05.993620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.993597    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 20:10:05.993739 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:05.993600    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-q5wzw\""
Jul 28 20:10:06.029638 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.029610    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"]
Jul 28 20:10:06.032784 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.032769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:06.041627 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.041605    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"]
Jul 28 20:10:06.103974 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.103898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wszr\" (UniqueName: \"kubernetes.io/projected/a75904b0-19db-4b4e-86c3-cf7312dd4d0d-kube-api-access-9wszr\") pod \"payload-pre-processing-687dbb6cf6-tgj9q\" (UID: \"a75904b0-19db-4b4e-86c3-cf7312dd4d0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:06.104122 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.103978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75904b0-19db-4b4e-86c3-cf7312dd4d0d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-tgj9q\" (UID: \"a75904b0-19db-4b4e-86c3-cf7312dd4d0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:06.104122 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.104021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q62z\" (UniqueName: \"kubernetes.io/projected/988d2ddd-e609-4125-8a53-b6904e158e7a-kube-api-access-5q62z\") pod \"payload-processing-5cc6cd444b-5gbgl\" (UID: \"988d2ddd-e609-4125-8a53-b6904e158e7a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:06.104122 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.104042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988d2ddd-e609-4125-8a53-b6904e158e7a-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5gbgl\" (UID: \"988d2ddd-e609-4125-8a53-b6904e158e7a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:06.205154 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.205119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75904b0-19db-4b4e-86c3-cf7312dd4d0d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-tgj9q\" (UID: \"a75904b0-19db-4b4e-86c3-cf7312dd4d0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:06.205370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.205184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q62z\" (UniqueName: \"kubernetes.io/projected/988d2ddd-e609-4125-8a53-b6904e158e7a-kube-api-access-5q62z\") pod \"payload-processing-5cc6cd444b-5gbgl\" (UID: \"988d2ddd-e609-4125-8a53-b6904e158e7a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:06.205370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.205220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988d2ddd-e609-4125-8a53-b6904e158e7a-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5gbgl\" (UID: \"988d2ddd-e609-4125-8a53-b6904e158e7a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:06.205370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.205251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wszr\" (UniqueName: \"kubernetes.io/projected/a75904b0-19db-4b4e-86c3-cf7312dd4d0d-kube-api-access-9wszr\") pod \"payload-pre-processing-687dbb6cf6-tgj9q\" (UID: \"a75904b0-19db-4b4e-86c3-cf7312dd4d0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:06.206112 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.206080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988d2ddd-e609-4125-8a53-b6904e158e7a-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5gbgl\" (UID: \"988d2ddd-e609-4125-8a53-b6904e158e7a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:06.206213 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.206080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75904b0-19db-4b4e-86c3-cf7312dd4d0d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-tgj9q\" (UID: \"a75904b0-19db-4b4e-86c3-cf7312dd4d0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:06.213243 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.213212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wszr\" (UniqueName: \"kubernetes.io/projected/a75904b0-19db-4b4e-86c3-cf7312dd4d0d-kube-api-access-9wszr\") pod \"payload-pre-processing-687dbb6cf6-tgj9q\" (UID: \"a75904b0-19db-4b4e-86c3-cf7312dd4d0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:06.213243 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.213219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q62z\" (UniqueName: \"kubernetes.io/projected/988d2ddd-e609-4125-8a53-b6904e158e7a-kube-api-access-5q62z\") pod \"payload-processing-5cc6cd444b-5gbgl\" (UID: \"988d2ddd-e609-4125-8a53-b6904e158e7a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:06.267130 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.267098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx" event={"ID":"d82f1086-427d-4a3c-ba1e-15aa78b38364","Type":"ContainerStarted","Data":"69ccc49d875f9f2b0e30e6ef52590cfb0f68dca8f59a9374cab2e035a742a8f7"}
Jul 28 20:10:06.302063 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.302023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:06.343880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.343841    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:06.441606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.441577    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5gbgl"]
Jul 28 20:10:06.443579 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:10:06.443542    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988d2ddd_e609_4125_8a53_b6904e158e7a.slice/crio-0ec309dae3aefc714d8df92e3db8bbf0fa2580cd015eaf7e387d5de1848af50c WatchSource:0}: Error finding container 0ec309dae3aefc714d8df92e3db8bbf0fa2580cd015eaf7e387d5de1848af50c: Status 404 returned error can't find the container with id 0ec309dae3aefc714d8df92e3db8bbf0fa2580cd015eaf7e387d5de1848af50c
Jul 28 20:10:06.492704 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:06.492678    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"]
Jul 28 20:10:06.494903 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:10:06.494875    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75904b0_19db_4b4e_86c3_cf7312dd4d0d.slice/crio-0aef89f41229eeae9fddc5ff09ad84dfd582e59c46e02832469106e183a34bc6 WatchSource:0}: Error finding container 0aef89f41229eeae9fddc5ff09ad84dfd582e59c46e02832469106e183a34bc6: Status 404 returned error can't find the container with id 0aef89f41229eeae9fddc5ff09ad84dfd582e59c46e02832469106e183a34bc6
Jul 28 20:10:07.276396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:07.276357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q" event={"ID":"a75904b0-19db-4b4e-86c3-cf7312dd4d0d","Type":"ContainerStarted","Data":"0aef89f41229eeae9fddc5ff09ad84dfd582e59c46e02832469106e183a34bc6"}
Jul 28 20:10:07.277962 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:07.277935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl" event={"ID":"988d2ddd-e609-4125-8a53-b6904e158e7a","Type":"ContainerStarted","Data":"0ec309dae3aefc714d8df92e3db8bbf0fa2580cd015eaf7e387d5de1848af50c"}
Jul 28 20:10:08.286338 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:08.286186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx" event={"ID":"d82f1086-427d-4a3c-ba1e-15aa78b38364","Type":"ContainerStarted","Data":"9d6d255aa02f912d60993919e8d926e28216051f91e1ca243c7e883454223874"}
Jul 28 20:10:08.286338 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:08.286328    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:08.307321 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:08.305160    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx" podStartSLOduration=1.257561661 podStartE2EDuration="3.305140662s" totalImagesPullingTime="2.047579001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:10:08.301033277 +0000 UTC m=+658.155344344" watchObservedRunningTime="2026-07-28 20:10:08.305140662 +0000 UTC m=+658.159451730"
Jul 28 20:10:10.299328 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:10.299273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl" event={"ID":"988d2ddd-e609-4125-8a53-b6904e158e7a","Type":"ContainerStarted","Data":"ed5560e5743d81e9ee0b9ec35d5873a1cea85d18b83fd185d399c28612e4ac5d"}
Jul 28 20:10:10.299841 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:10.299347    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:10.300846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:10.300824    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q" event={"ID":"a75904b0-19db-4b4e-86c3-cf7312dd4d0d","Type":"ContainerStarted","Data":"9b9f5fed767ab6a398effff22ae65d6ff42874cec0ae42db44e3a0281ef85eef"}
Jul 28 20:10:10.300980 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:10.300889    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:10.319479 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:10.319433    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl" podStartSLOduration=2.289191745 podStartE2EDuration="5.319417907s" totalImagesPullingTime="3.030226162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:10:10.317401739 +0000 UTC m=+660.171712815" watchObservedRunningTime="2026-07-28 20:10:10.319417907 +0000 UTC m=+660.173728973"
Jul 28 20:10:10.335541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:10.335475    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q" podStartSLOduration=1.356575136 podStartE2EDuration="4.335455555s" totalImagesPullingTime="2.978880419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:10:10.333152664 +0000 UTC m=+660.187463755" watchObservedRunningTime="2026-07-28 20:10:10.335455555 +0000 UTC m=+660.189766623"
Jul 28 20:10:14.297187 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:14.297152    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74bb79f4fd-7kzlx"
Jul 28 20:10:14.320347 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:14.320284    2582 generic.go:356] "Generic (PLEG): container finished" podID="2d099fda-4a87-45cb-a88f-a1d546cf1e6e" containerID="386f9a5928689f9017b11e3aa9f612f113128db4e459824552c2d106cecb5cda" exitCode=0
Jul 28 20:10:14.320497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:14.320345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z" event={"ID":"2d099fda-4a87-45cb-a88f-a1d546cf1e6e","Type":"ContainerDied","Data":"386f9a5928689f9017b11e3aa9f612f113128db4e459824552c2d106cecb5cda"}
Jul 28 20:10:14.320773 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:14.320751    2582 scope.go:117] "RemoveContainer" containerID="386f9a5928689f9017b11e3aa9f612f113128db4e459824552c2d106cecb5cda"
Jul 28 20:10:14.321808 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:14.321784    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:10:14.321878 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:14.321843    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:10:14.321878 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:14.321873    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:10:15.330002 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:15.329930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z" event={"ID":"2d099fda-4a87-45cb-a88f-a1d546cf1e6e","Type":"ContainerStarted","Data":"b67b18481e35321994209badf0eda610724491a332a5fabd315f2256a93ae12d"}
Jul 28 20:10:16.006972 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:16.006928    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:16.105256 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:16.105223    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:16.290816 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:16.290714    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:10:16.291939 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:16.291914    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-c5q2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 28 20:10:16.292058 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:16.291960    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z" podUID="2d099fda-4a87-45cb-a88f-a1d546cf1e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 28 20:10:17.291706 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:17.291670    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-c5q2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 28 20:10:17.292101 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:17.291725    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z" podUID="2d099fda-4a87-45cb-a88f-a1d546cf1e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 28 20:10:17.887825 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:17.887794    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:10:18.291436 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:18.291395    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-c5q2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 28 20:10:18.291613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:18.291473    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z" podUID="2d099fda-4a87-45cb-a88f-a1d546cf1e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 28 20:10:19.295970 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:19.295942    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:10:19.297037 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:19.297017    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-c5q2z"
Jul 28 20:10:21.308090 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:21.308062    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5gbgl"
Jul 28 20:10:21.308517 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:21.308126    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tgj9q"
Jul 28 20:10:22.889138 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:22.889099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:44.191886 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.191853    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"]
Jul 28 20:10:44.198692 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.198669    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.201537 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.201515    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rnpl5\""
Jul 28 20:10:44.201974 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.201956    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 20:10:44.202488 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.202473    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 20:10:44.202642 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.202510    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 28 20:10:44.208422 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.208400    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"]
Jul 28 20:10:44.217585 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.217562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:44.248848 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.248815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.248995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.248864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.248995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.248924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52d9s\" (UniqueName: \"kubernetes.io/projected/095c3f09-0f14-4b09-858e-2dc3202c3c47-kube-api-access-52d9s\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.248995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.248966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.248995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.248987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095c3f09-0f14-4b09-858e-2dc3202c3c47-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.249129 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.249049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.349613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.349577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.349778 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.349626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.349778 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.349674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52d9s\" (UniqueName: \"kubernetes.io/projected/095c3f09-0f14-4b09-858e-2dc3202c3c47-kube-api-access-52d9s\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.349778 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.349722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.349778 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.349748    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095c3f09-0f14-4b09-858e-2dc3202c3c47-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.349974 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.349800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.350057 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.350035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.350148 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.350100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.357404 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.357371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.357909 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.357874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095c3f09-0f14-4b09-858e-2dc3202c3c47-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.358570 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.358543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095c3f09-0f14-4b09-858e-2dc3202c3c47-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.358683 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.358615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52d9s\" (UniqueName: \"kubernetes.io/projected/095c3f09-0f14-4b09-858e-2dc3202c3c47-kube-api-access-52d9s\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs\" (UID: \"095c3f09-0f14-4b09-858e-2dc3202c3c47\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.509662 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.509622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:44.642562 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.642523    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"]
Jul 28 20:10:44.643926 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:10:44.643894    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095c3f09_0f14_4b09_858e_2dc3202c3c47.slice/crio-7e80985d7f3a10b3acb2b1a8df75d242895d8027ab65c8e44b2426de8e725e97 WatchSource:0}: Error finding container 7e80985d7f3a10b3acb2b1a8df75d242895d8027ab65c8e44b2426de8e725e97: Status 404 returned error can't find the container with id 7e80985d7f3a10b3acb2b1a8df75d242895d8027ab65c8e44b2426de8e725e97
Jul 28 20:10:44.645447 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:44.645427    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:10:45.471987 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:45.471955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs" event={"ID":"095c3f09-0f14-4b09-858e-2dc3202c3c47","Type":"ContainerStarted","Data":"7e80985d7f3a10b3acb2b1a8df75d242895d8027ab65c8e44b2426de8e725e97"}
Jul 28 20:10:46.477760 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:46.477720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs" event={"ID":"095c3f09-0f14-4b09-858e-2dc3202c3c47","Type":"ContainerStarted","Data":"393fd2ec699b31ebd181e72022fe230287ed76a14523a571aca8b90e1fb14888"}
Jul 28 20:10:46.496065 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:46.496013    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs" podStartSLOduration=1.32153429 podStartE2EDuration="2.495997879s" totalImagesPullingTime="1.174463589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:10:46.494389029 +0000 UTC m=+696.348700098" watchObservedRunningTime="2026-07-28 20:10:46.495997879 +0000 UTC m=+696.350308944"
Jul 28 20:10:54.509817 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:54.509774    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:54.510251 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:54.509838    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:54.522836 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:54.522812    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:55.529761 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:55.529733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs"
Jul 28 20:10:57.509467 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:10:57.509428    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:05.299138 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:05.299097    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:14.677651 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.677619    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"]
Jul 28 20:11:14.681214 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.681194    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.683620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.683599    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 28 20:11:14.693933 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.693903    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"]
Jul 28 20:11:14.694867 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.694846    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:14.817844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.817802    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.818033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.817863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.818033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.817906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.818033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.817961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.818033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.817995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d59kz\" (UniqueName: \"kubernetes.io/projected/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-kube-api-access-d59kz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.818033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.818025    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.919448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.919415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.919448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.919448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.919711 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.919480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.919711 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.919663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59kz\" (UniqueName: \"kubernetes.io/projected/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-kube-api-access-d59kz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.919830 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.919711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.919830 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.919751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.919926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.919869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.919926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.919887    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.920097 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.920072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.921845 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.921823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.922136 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.922111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.927594 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.927573    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59kz\" (UniqueName: \"kubernetes.io/projected/2f0bd08e-5bb4-4c0b-9872-4aa08fef944f-kube-api-access-d59kz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx\" (UID: \"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:14.994647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:14.994620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:15.131577 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:15.131551    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"]
Jul 28 20:11:15.611048 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:15.610965    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx" event={"ID":"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f","Type":"ContainerStarted","Data":"abaa3e33dffaf76eeae2ca0890c155cef2c5fce61d27680a8ceaf172a4b49d83"}
Jul 28 20:11:15.611048 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:15.611016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx" event={"ID":"2f0bd08e-5bb4-4c0b-9872-4aa08fef944f","Type":"ContainerStarted","Data":"5e4423036d1da58fdb7940eca1230f9b7b94418c775513169dfe2ddce8ff3852"}
Jul 28 20:11:15.628748 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:15.628682    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx" podStartSLOduration=1.449191691 podStartE2EDuration="1.628661481s" totalImagesPullingTime="179.46979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:11:15.628548535 +0000 UTC m=+725.482859602" watchObservedRunningTime="2026-07-28 20:11:15.628661481 +0000 UTC m=+725.482972548"
Jul 28 20:11:24.995161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:24.995124    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:24.995665 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:24.995189    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:25.007761 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:25.007732    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:25.668094 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:25.668065    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx"
Jul 28 20:11:40.731149 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:40.731114    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:54.218326 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:11:54.218271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:32.034336 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:32.034304    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:32.135197 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:32.135156    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:54.740835 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:54.740799    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-ddc577dd-92px5"]
Jul 28 20:12:54.744504 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:54.744480    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ddc577dd-92px5"
Jul 28 20:12:54.753357 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:54.753332    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ddc577dd-92px5"]
Jul 28 20:12:54.802933 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:54.802894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ab1c29d-5e53-4efe-bf02-a51a29d71f95-tls-cert\") pod \"authorino-ddc577dd-92px5\" (UID: \"9ab1c29d-5e53-4efe-bf02-a51a29d71f95\") " pod="kuadrant-system/authorino-ddc577dd-92px5"
Jul 28 20:12:54.803105 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:54.802973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjr5\" (UniqueName: \"kubernetes.io/projected/9ab1c29d-5e53-4efe-bf02-a51a29d71f95-kube-api-access-kzjr5\") pod \"authorino-ddc577dd-92px5\" (UID: \"9ab1c29d-5e53-4efe-bf02-a51a29d71f95\") " pod="kuadrant-system/authorino-ddc577dd-92px5"
Jul 28 20:12:54.904203 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:54.904158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjr5\" (UniqueName: \"kubernetes.io/projected/9ab1c29d-5e53-4efe-bf02-a51a29d71f95-kube-api-access-kzjr5\") pod \"authorino-ddc577dd-92px5\" (UID: \"9ab1c29d-5e53-4efe-bf02-a51a29d71f95\") " pod="kuadrant-system/authorino-ddc577dd-92px5"
Jul 28 20:12:54.904400 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:54.904261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ab1c29d-5e53-4efe-bf02-a51a29d71f95-tls-cert\") pod \"authorino-ddc577dd-92px5\" (UID: \"9ab1c29d-5e53-4efe-bf02-a51a29d71f95\") " pod="kuadrant-system/authorino-ddc577dd-92px5"
Jul 28 20:12:54.906699 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:54.906667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ab1c29d-5e53-4efe-bf02-a51a29d71f95-tls-cert\") pod \"authorino-ddc577dd-92px5\" (UID: \"9ab1c29d-5e53-4efe-bf02-a51a29d71f95\") " pod="kuadrant-system/authorino-ddc577dd-92px5"
Jul 28 20:12:54.911970 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:54.911942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjr5\" (UniqueName: \"kubernetes.io/projected/9ab1c29d-5e53-4efe-bf02-a51a29d71f95-kube-api-access-kzjr5\") pod \"authorino-ddc577dd-92px5\" (UID: \"9ab1c29d-5e53-4efe-bf02-a51a29d71f95\") " pod="kuadrant-system/authorino-ddc577dd-92px5"
Jul 28 20:12:55.054548 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:55.054459    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ddc577dd-92px5"
Jul 28 20:12:55.136573 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:55.136535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:55.196550 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:55.196526    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ddc577dd-92px5"]
Jul 28 20:12:55.198691 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:12:55.198663    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ab1c29d_5e53_4efe_bf02_a51a29d71f95.slice/crio-dad7394ff8bd29623be47611249a609012cc4646fdfca95e380d4dea897d368f WatchSource:0}: Error finding container dad7394ff8bd29623be47611249a609012cc4646fdfca95e380d4dea897d368f: Status 404 returned error can't find the container with id dad7394ff8bd29623be47611249a609012cc4646fdfca95e380d4dea897d368f
Jul 28 20:12:55.238021 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:55.237991    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:56.087645 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.087609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ddc577dd-92px5" event={"ID":"9ab1c29d-5e53-4efe-bf02-a51a29d71f95","Type":"ContainerStarted","Data":"5f88d3b495cdbe5b118810229af62ed827450b05ca83cd0a2c4f55d7b5c56c5a"}
Jul 28 20:12:56.087645 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.087655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ddc577dd-92px5" event={"ID":"9ab1c29d-5e53-4efe-bf02-a51a29d71f95","Type":"ContainerStarted","Data":"dad7394ff8bd29623be47611249a609012cc4646fdfca95e380d4dea897d368f"}
Jul 28 20:12:56.103745 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.103682    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-ddc577dd-92px5" podStartSLOduration=1.7157164630000001 podStartE2EDuration="2.103662467s" totalImagesPullingTime="387.946004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:12:56.10349457 +0000 UTC m=+825.957805637" watchObservedRunningTime="2026-07-28 20:12:56.103662467 +0000 UTC m=+825.957973533"
Jul 28 20:12:56.132285 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.131871    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55cbc584f7-h8vbj"]
Jul 28 20:12:56.132285 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.132087    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-55cbc584f7-h8vbj" podUID="8661cb26-3520-4297-8d93-66a4e1395d57" containerName="authorino" containerID="cri-o://1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28" gracePeriod=30
Jul 28 20:12:56.330609 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.330534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:56.415983 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.415953    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:12:56.437541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.437501    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:56.519806 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.519770    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28dgv\" (UniqueName: \"kubernetes.io/projected/8661cb26-3520-4297-8d93-66a4e1395d57-kube-api-access-28dgv\") pod \"8661cb26-3520-4297-8d93-66a4e1395d57\" (UID: \"8661cb26-3520-4297-8d93-66a4e1395d57\") "
Jul 28 20:12:56.519993 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.519876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8661cb26-3520-4297-8d93-66a4e1395d57-tls-cert\") pod \"8661cb26-3520-4297-8d93-66a4e1395d57\" (UID: \"8661cb26-3520-4297-8d93-66a4e1395d57\") "
Jul 28 20:12:56.521922 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.521887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8661cb26-3520-4297-8d93-66a4e1395d57-kube-api-access-28dgv" (OuterVolumeSpecName: "kube-api-access-28dgv") pod "8661cb26-3520-4297-8d93-66a4e1395d57" (UID: "8661cb26-3520-4297-8d93-66a4e1395d57"). InnerVolumeSpecName "kube-api-access-28dgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:12:56.530636 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.530610    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8661cb26-3520-4297-8d93-66a4e1395d57-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8661cb26-3520-4297-8d93-66a4e1395d57" (UID: "8661cb26-3520-4297-8d93-66a4e1395d57"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:12:56.620879 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.620836    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28dgv\" (UniqueName: \"kubernetes.io/projected/8661cb26-3520-4297-8d93-66a4e1395d57-kube-api-access-28dgv\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:12:56.620879 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:56.620872    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8661cb26-3520-4297-8d93-66a4e1395d57-tls-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:12:57.095877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:57.095838    2582 generic.go:356] "Generic (PLEG): container finished" podID="8661cb26-3520-4297-8d93-66a4e1395d57" containerID="1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28" exitCode=0
Jul 28 20:12:57.095877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:57.095879    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55cbc584f7-h8vbj"
Jul 28 20:12:57.096416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:57.095920    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55cbc584f7-h8vbj" event={"ID":"8661cb26-3520-4297-8d93-66a4e1395d57","Type":"ContainerDied","Data":"1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28"}
Jul 28 20:12:57.096416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:57.095960    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55cbc584f7-h8vbj" event={"ID":"8661cb26-3520-4297-8d93-66a4e1395d57","Type":"ContainerDied","Data":"263fcc6733da6bb06f1eb2702573dc6ddeb9155301424fe0e19f9a90861c2818"}
Jul 28 20:12:57.096416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:57.095984    2582 scope.go:117] "RemoveContainer" containerID="1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28"
Jul 28 20:12:57.105950 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:57.105928    2582 scope.go:117] "RemoveContainer" containerID="1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28"
Jul 28 20:12:57.106257 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:12:57.106232    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28\": container with ID starting with 1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28 not found: ID does not exist" containerID="1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28"
Jul 28 20:12:57.106345 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:57.106262    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28"} err="failed to get container status \"1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28\": rpc error: code = NotFound desc = could not find container \"1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28\": container with ID starting with 1a58e3b4bc9db2e3f1529330c987515890c4af19401c7cdd1a5cc1a7295e7a28 not found: ID does not exist"
Jul 28 20:12:57.116968 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:57.116941    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55cbc584f7-h8vbj"]
Jul 28 20:12:57.123087 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:57.123061    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55cbc584f7-h8vbj"]
Jul 28 20:12:58.583212 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:12:58.583177    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8661cb26-3520-4297-8d93-66a4e1395d57" path="/var/lib/kubelet/pods/8661cb26-3520-4297-8d93-66a4e1395d57/volumes"
Jul 28 20:13:11.440510 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:11.440476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:29.386505 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:29.386468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:42.901469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:42.901426    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58546478d9-tzj6c"]
Jul 28 20:13:42.901984 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:42.901700    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58546478d9-tzj6c" podUID="2846dc92-55a8-4be0-8906-6233c33d212c" containerName="manager" containerID="cri-o://9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4" gracePeriod=30
Jul 28 20:13:43.153070 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.153012    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:13:43.227363 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.227324    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbzf8\" (UniqueName: \"kubernetes.io/projected/2846dc92-55a8-4be0-8906-6233c33d212c-kube-api-access-xbzf8\") pod \"2846dc92-55a8-4be0-8906-6233c33d212c\" (UID: \"2846dc92-55a8-4be0-8906-6233c33d212c\") "
Jul 28 20:13:43.227555 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.227394    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2846dc92-55a8-4be0-8906-6233c33d212c-webhook-cert\") pod \"2846dc92-55a8-4be0-8906-6233c33d212c\" (UID: \"2846dc92-55a8-4be0-8906-6233c33d212c\") "
Jul 28 20:13:43.229453 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.229420    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2846dc92-55a8-4be0-8906-6233c33d212c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2846dc92-55a8-4be0-8906-6233c33d212c" (UID: "2846dc92-55a8-4be0-8906-6233c33d212c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:13:43.229562 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.229468    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2846dc92-55a8-4be0-8906-6233c33d212c-kube-api-access-xbzf8" (OuterVolumeSpecName: "kube-api-access-xbzf8") pod "2846dc92-55a8-4be0-8906-6233c33d212c" (UID: "2846dc92-55a8-4be0-8906-6233c33d212c"). InnerVolumeSpecName "kube-api-access-xbzf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:13:43.316837 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.316791    2582 generic.go:356] "Generic (PLEG): container finished" podID="2846dc92-55a8-4be0-8906-6233c33d212c" containerID="9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4" exitCode=0
Jul 28 20:13:43.316837 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.316839    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-tzj6c"
Jul 28 20:13:43.317097 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.316864    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-tzj6c" event={"ID":"2846dc92-55a8-4be0-8906-6233c33d212c","Type":"ContainerDied","Data":"9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4"}
Jul 28 20:13:43.317097 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.316932    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-tzj6c" event={"ID":"2846dc92-55a8-4be0-8906-6233c33d212c","Type":"ContainerDied","Data":"1351920a13b0215e7a0f6d518669ef30ae59120ec4af45c9299ec034c0b27e50"}
Jul 28 20:13:43.317097 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.316957    2582 scope.go:117] "RemoveContainer" containerID="9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4"
Jul 28 20:13:43.328386 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.328361    2582 scope.go:117] "RemoveContainer" containerID="9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4"
Jul 28 20:13:43.328880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.328712    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2846dc92-55a8-4be0-8906-6233c33d212c-webhook-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:13:43.328880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.328734    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbzf8\" (UniqueName: \"kubernetes.io/projected/2846dc92-55a8-4be0-8906-6233c33d212c-kube-api-access-xbzf8\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:13:43.328880 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:13:43.328760    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4\": container with ID starting with 9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4 not found: ID does not exist" containerID="9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4"
Jul 28 20:13:43.328880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.328796    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4"} err="failed to get container status \"9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4\": rpc error: code = NotFound desc = could not find container \"9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4\": container with ID starting with 9ff63343b2b715eb82d31f9a7f186422fb37ba99d1b45d824b3cf1c67fd302a4 not found: ID does not exist"
Jul 28 20:13:43.342834 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.342812    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58546478d9-tzj6c"]
Jul 28 20:13:43.344423 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:43.344404    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58546478d9-tzj6c"]
Jul 28 20:13:44.582743 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:13:44.582710    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2846dc92-55a8-4be0-8906-6233c33d212c" path="/var/lib/kubelet/pods/2846dc92-55a8-4be0-8906-6233c33d212c/volumes"
Jul 28 20:14:28.320845 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:28.320814    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:29.452125 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:29.452091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:44.098500 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.098468    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58546478d9-wdgbj"]
Jul 28 20:14:44.098981 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.098812    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8661cb26-3520-4297-8d93-66a4e1395d57" containerName="authorino"
Jul 28 20:14:44.098981 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.098822    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8661cb26-3520-4297-8d93-66a4e1395d57" containerName="authorino"
Jul 28 20:14:44.098981 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.098834    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2846dc92-55a8-4be0-8906-6233c33d212c" containerName="manager"
Jul 28 20:14:44.098981 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.098840    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2846dc92-55a8-4be0-8906-6233c33d212c" containerName="manager"
Jul 28 20:14:44.098981 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.098904    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2846dc92-55a8-4be0-8906-6233c33d212c" containerName="manager"
Jul 28 20:14:44.098981 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.098914    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8661cb26-3520-4297-8d93-66a4e1395d57" containerName="authorino"
Jul 28 20:14:44.102058 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.102037    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:14:44.105608 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.105586    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vc59v\""
Jul 28 20:14:44.105733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.105645    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:14:44.105733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.105677    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:14:44.111228 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.111206    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58546478d9-wdgbj"]
Jul 28 20:14:44.180664 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.180623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42zgb\" (UniqueName: \"kubernetes.io/projected/548bcfff-0357-41a8-aa7e-70da8752626a-kube-api-access-42zgb\") pod \"maas-controller-58546478d9-wdgbj\" (UID: \"548bcfff-0357-41a8-aa7e-70da8752626a\") " pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:14:44.180869 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.180689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/548bcfff-0357-41a8-aa7e-70da8752626a-webhook-cert\") pod \"maas-controller-58546478d9-wdgbj\" (UID: \"548bcfff-0357-41a8-aa7e-70da8752626a\") " pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:14:44.282041 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.281944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/548bcfff-0357-41a8-aa7e-70da8752626a-webhook-cert\") pod \"maas-controller-58546478d9-wdgbj\" (UID: \"548bcfff-0357-41a8-aa7e-70da8752626a\") " pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:14:44.282211 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.282156    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zgb\" (UniqueName: \"kubernetes.io/projected/548bcfff-0357-41a8-aa7e-70da8752626a-kube-api-access-42zgb\") pod \"maas-controller-58546478d9-wdgbj\" (UID: \"548bcfff-0357-41a8-aa7e-70da8752626a\") " pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:14:44.284263 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.284241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/548bcfff-0357-41a8-aa7e-70da8752626a-webhook-cert\") pod \"maas-controller-58546478d9-wdgbj\" (UID: \"548bcfff-0357-41a8-aa7e-70da8752626a\") " pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:14:44.289893 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.289866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zgb\" (UniqueName: \"kubernetes.io/projected/548bcfff-0357-41a8-aa7e-70da8752626a-kube-api-access-42zgb\") pod \"maas-controller-58546478d9-wdgbj\" (UID: \"548bcfff-0357-41a8-aa7e-70da8752626a\") " pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:14:44.412664 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.412556    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:14:44.548166 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.548139    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58546478d9-wdgbj"]
Jul 28 20:14:44.551090 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:14:44.551064    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548bcfff_0357_41a8_aa7e_70da8752626a.slice/crio-1b27a5742f8e9cb5cd2a131c890883375bf6af22e4f2695e5704c6133a5be3da WatchSource:0}: Error finding container 1b27a5742f8e9cb5cd2a131c890883375bf6af22e4f2695e5704c6133a5be3da: Status 404 returned error can't find the container with id 1b27a5742f8e9cb5cd2a131c890883375bf6af22e4f2695e5704c6133a5be3da
Jul 28 20:14:44.596711 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:44.596675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-wdgbj" event={"ID":"548bcfff-0357-41a8-aa7e-70da8752626a","Type":"ContainerStarted","Data":"1b27a5742f8e9cb5cd2a131c890883375bf6af22e4f2695e5704c6133a5be3da"}
Jul 28 20:14:45.603621 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:45.603583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-wdgbj" event={"ID":"548bcfff-0357-41a8-aa7e-70da8752626a","Type":"ContainerStarted","Data":"eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d"}
Jul 28 20:14:45.604025 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:45.603643    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:14:45.622162 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:45.622109    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58546478d9-wdgbj" podStartSLOduration=1.13227517 podStartE2EDuration="1.622091675s" totalImagesPullingTime="489.816505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:45.619757322 +0000 UTC m=+935.474068388" watchObservedRunningTime="2026-07-28 20:14:45.622091675 +0000 UTC m=+935.476402742"
Jul 28 20:14:56.610941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:14:56.610900    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:15:01.165051 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.165019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:01.262994 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.262963    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58546478d9-wdgbj"]
Jul 28 20:15:01.263178 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.263153    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58546478d9-wdgbj" podUID="548bcfff-0357-41a8-aa7e-70da8752626a" containerName="manager" containerID="cri-o://eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d" gracePeriod=30
Jul 28 20:15:01.509845 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.509817    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:15:01.536323 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.536270    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42zgb\" (UniqueName: \"kubernetes.io/projected/548bcfff-0357-41a8-aa7e-70da8752626a-kube-api-access-42zgb\") pod \"548bcfff-0357-41a8-aa7e-70da8752626a\" (UID: \"548bcfff-0357-41a8-aa7e-70da8752626a\") "
Jul 28 20:15:01.536598 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.536578    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/548bcfff-0357-41a8-aa7e-70da8752626a-webhook-cert\") pod \"548bcfff-0357-41a8-aa7e-70da8752626a\" (UID: \"548bcfff-0357-41a8-aa7e-70da8752626a\") "
Jul 28 20:15:01.538712 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.538654    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548bcfff-0357-41a8-aa7e-70da8752626a-kube-api-access-42zgb" (OuterVolumeSpecName: "kube-api-access-42zgb") pod "548bcfff-0357-41a8-aa7e-70da8752626a" (UID: "548bcfff-0357-41a8-aa7e-70da8752626a"). InnerVolumeSpecName "kube-api-access-42zgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:15:01.539052 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.539024    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548bcfff-0357-41a8-aa7e-70da8752626a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "548bcfff-0357-41a8-aa7e-70da8752626a" (UID: "548bcfff-0357-41a8-aa7e-70da8752626a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:15:01.637709 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.637667    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/548bcfff-0357-41a8-aa7e-70da8752626a-webhook-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:15:01.637709 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.637700    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42zgb\" (UniqueName: \"kubernetes.io/projected/548bcfff-0357-41a8-aa7e-70da8752626a-kube-api-access-42zgb\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:15:01.680861 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.680764    2582 generic.go:356] "Generic (PLEG): container finished" podID="548bcfff-0357-41a8-aa7e-70da8752626a" containerID="eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d" exitCode=0
Jul 28 20:15:01.680861 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.680807    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-wdgbj"
Jul 28 20:15:01.681056 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.680862    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-wdgbj" event={"ID":"548bcfff-0357-41a8-aa7e-70da8752626a","Type":"ContainerDied","Data":"eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d"}
Jul 28 20:15:01.681056 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.680916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-wdgbj" event={"ID":"548bcfff-0357-41a8-aa7e-70da8752626a","Type":"ContainerDied","Data":"1b27a5742f8e9cb5cd2a131c890883375bf6af22e4f2695e5704c6133a5be3da"}
Jul 28 20:15:01.681056 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.680937    2582 scope.go:117] "RemoveContainer" containerID="eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d"
Jul 28 20:15:01.689933 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.689915    2582 scope.go:117] "RemoveContainer" containerID="eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d"
Jul 28 20:15:01.690189 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:15:01.690171    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d\": container with ID starting with eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d not found: ID does not exist" containerID="eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d"
Jul 28 20:15:01.690241 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.690197    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d"} err="failed to get container status \"eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d\": rpc error: code = NotFound desc = could not find container \"eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d\": container with ID starting with eed42b1bc1c9bcc1cbfbf7a68fc14415be12fec840192f6c937dce3dc4cbde2d not found: ID does not exist"
Jul 28 20:15:01.702391 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.702365    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58546478d9-wdgbj"]
Jul 28 20:15:01.706191 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:01.706166    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58546478d9-wdgbj"]
Jul 28 20:15:02.413068 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.413030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:02.475893 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.475859    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58546478d9-bjkcp"]
Jul 28 20:15:02.476222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.476210    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548bcfff-0357-41a8-aa7e-70da8752626a" containerName="manager"
Jul 28 20:15:02.476311 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.476224    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="548bcfff-0357-41a8-aa7e-70da8752626a" containerName="manager"
Jul 28 20:15:02.476311 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.476308    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="548bcfff-0357-41a8-aa7e-70da8752626a" containerName="manager"
Jul 28 20:15:02.480530 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.480509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:02.483827 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.483807    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vc59v\""
Jul 28 20:15:02.483827 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.483824    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:15:02.484001 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.483807    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:15:02.487851 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.487630    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58546478d9-bjkcp"]
Jul 28 20:15:02.544338 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.544277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhk92\" (UniqueName: \"kubernetes.io/projected/c0d06697-ef11-4db9-bed7-95b1e394199f-kube-api-access-vhk92\") pod \"maas-controller-58546478d9-bjkcp\" (UID: \"c0d06697-ef11-4db9-bed7-95b1e394199f\") " pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:02.544507 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.544415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0d06697-ef11-4db9-bed7-95b1e394199f-webhook-cert\") pod \"maas-controller-58546478d9-bjkcp\" (UID: \"c0d06697-ef11-4db9-bed7-95b1e394199f\") " pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:02.583025 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.582986    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548bcfff-0357-41a8-aa7e-70da8752626a" path="/var/lib/kubelet/pods/548bcfff-0357-41a8-aa7e-70da8752626a/volumes"
Jul 28 20:15:02.645554 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.645517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0d06697-ef11-4db9-bed7-95b1e394199f-webhook-cert\") pod \"maas-controller-58546478d9-bjkcp\" (UID: \"c0d06697-ef11-4db9-bed7-95b1e394199f\") " pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:02.645717 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.645569    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhk92\" (UniqueName: \"kubernetes.io/projected/c0d06697-ef11-4db9-bed7-95b1e394199f-kube-api-access-vhk92\") pod \"maas-controller-58546478d9-bjkcp\" (UID: \"c0d06697-ef11-4db9-bed7-95b1e394199f\") " pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:02.647887 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.647864    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0d06697-ef11-4db9-bed7-95b1e394199f-webhook-cert\") pod \"maas-controller-58546478d9-bjkcp\" (UID: \"c0d06697-ef11-4db9-bed7-95b1e394199f\") " pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:02.653428 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.653406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhk92\" (UniqueName: \"kubernetes.io/projected/c0d06697-ef11-4db9-bed7-95b1e394199f-kube-api-access-vhk92\") pod \"maas-controller-58546478d9-bjkcp\" (UID: \"c0d06697-ef11-4db9-bed7-95b1e394199f\") " pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:02.792387 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.792356    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:02.921762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:02.921733    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58546478d9-bjkcp"]
Jul 28 20:15:02.923141 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:15:02.923115    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d06697_ef11_4db9_bed7_95b1e394199f.slice/crio-34417e74020626c86bfe8739707605ecaf918c8da00dfcbb12dc9d96bdfed114 WatchSource:0}: Error finding container 34417e74020626c86bfe8739707605ecaf918c8da00dfcbb12dc9d96bdfed114: Status 404 returned error can't find the container with id 34417e74020626c86bfe8739707605ecaf918c8da00dfcbb12dc9d96bdfed114
Jul 28 20:15:03.692599 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:03.692510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-bjkcp" event={"ID":"c0d06697-ef11-4db9-bed7-95b1e394199f","Type":"ContainerStarted","Data":"7009d8c9e2cc02670cdeda95b9bf96450fd8e5ac616b938b5536e3b052c1c6e9"}
Jul 28 20:15:03.692599 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:03.692559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58546478d9-bjkcp" event={"ID":"c0d06697-ef11-4db9-bed7-95b1e394199f","Type":"ContainerStarted","Data":"34417e74020626c86bfe8739707605ecaf918c8da00dfcbb12dc9d96bdfed114"}
Jul 28 20:15:03.693093 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:03.692650    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:03.709171 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:03.709123    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58546478d9-bjkcp" podStartSLOduration=1.279391193 podStartE2EDuration="1.709108084s" totalImagesPullingTime="429.716891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:03.707702745 +0000 UTC m=+953.562013811" watchObservedRunningTime="2026-07-28 20:15:03.709108084 +0000 UTC m=+953.563419149"
Jul 28 20:15:14.699978 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:14.699946    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58546478d9-bjkcp"
Jul 28 20:15:34.572721 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:34.572629    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:34.668041 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:34.668009    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:34.790510 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:34.790474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:40.570406 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:40.570369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:40.673856 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:40.673822    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:40.770124 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:40.770090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:40.925689 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:40.925604    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:46.691946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:46.691891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:46.790914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:15:46.790877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:47.189792 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:18:47.189704    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:47.280578 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:18:47.280536    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:50.713160 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:18:50.713128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:20:02.922642 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:02.922556    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"]
Jul 28 20:20:02.923197 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:02.922767    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt" podUID="9fc94170-3847-4197-be1d-730c927c39ae" containerName="manager" containerID="cri-o://fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013" gracePeriod=10
Jul 28 20:20:05.878196 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:05.878173    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:20:05.944440 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:05.944411    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztph6\" (UniqueName: \"kubernetes.io/projected/9fc94170-3847-4197-be1d-730c927c39ae-kube-api-access-ztph6\") pod \"9fc94170-3847-4197-be1d-730c927c39ae\" (UID: \"9fc94170-3847-4197-be1d-730c927c39ae\") "
Jul 28 20:20:05.944623 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:05.944510    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc94170-3847-4197-be1d-730c927c39ae-extensions-socket-volume\") pod \"9fc94170-3847-4197-be1d-730c927c39ae\" (UID: \"9fc94170-3847-4197-be1d-730c927c39ae\") "
Jul 28 20:20:05.944860 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:05.944834    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc94170-3847-4197-be1d-730c927c39ae-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9fc94170-3847-4197-be1d-730c927c39ae" (UID: "9fc94170-3847-4197-be1d-730c927c39ae"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:20:05.946436 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:05.946405    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc94170-3847-4197-be1d-730c927c39ae-kube-api-access-ztph6" (OuterVolumeSpecName: "kube-api-access-ztph6") pod "9fc94170-3847-4197-be1d-730c927c39ae" (UID: "9fc94170-3847-4197-be1d-730c927c39ae"). InnerVolumeSpecName "kube-api-access-ztph6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:20:06.045816 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.045783    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztph6\" (UniqueName: \"kubernetes.io/projected/9fc94170-3847-4197-be1d-730c927c39ae-kube-api-access-ztph6\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:20:06.045816 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.045810    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc94170-3847-4197-be1d-730c927c39ae-extensions-socket-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:20:06.123756 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.123717    2582 generic.go:356] "Generic (PLEG): container finished" podID="9fc94170-3847-4197-be1d-730c927c39ae" containerID="fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013" exitCode=0
Jul 28 20:20:06.123947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.123761    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"
Jul 28 20:20:06.123947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.123809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt" event={"ID":"9fc94170-3847-4197-be1d-730c927c39ae","Type":"ContainerDied","Data":"fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013"}
Jul 28 20:20:06.123947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.123870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt" event={"ID":"9fc94170-3847-4197-be1d-730c927c39ae","Type":"ContainerDied","Data":"f3c7eecd363faf47d505f0f26b1cecb0d779716808db39ee468c8849ed36091d"}
Jul 28 20:20:06.123947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.123901    2582 scope.go:117] "RemoveContainer" containerID="fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013"
Jul 28 20:20:06.133601 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.133582    2582 scope.go:117] "RemoveContainer" containerID="fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013"
Jul 28 20:20:06.133853 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:20:06.133836    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013\": container with ID starting with fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013 not found: ID does not exist" containerID="fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013"
Jul 28 20:20:06.133905 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.133862    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013"} err="failed to get container status \"fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013\": rpc error: code = NotFound desc = could not find container \"fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013\": container with ID starting with fb1abf7c9ca46dee3ac40ce63a1813eef9c000b9d68befdbcd81fe926318f013 not found: ID does not exist"
Jul 28 20:20:06.146424 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.146396    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"]
Jul 28 20:20:06.150473 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.150453    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chtqt"]
Jul 28 20:20:06.583774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:06.583734    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc94170-3847-4197-be1d-730c927c39ae" path="/var/lib/kubelet/pods/9fc94170-3847-4197-be1d-730c927c39ae/volumes"
Jul 28 20:20:08.427465 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:08.427428    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:20:44.455083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:20:44.455043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:21:04.066953 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.066921    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"]
Jul 28 20:21:04.067380 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.067331    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc94170-3847-4197-be1d-730c927c39ae" containerName="manager"
Jul 28 20:21:04.067380 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.067345    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc94170-3847-4197-be1d-730c927c39ae" containerName="manager"
Jul 28 20:21:04.067471 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.067406    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc94170-3847-4197-be1d-730c927c39ae" containerName="manager"
Jul 28 20:21:04.070432 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.070417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:21:04.077303 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.077270    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jrtxx\""
Jul 28 20:21:04.103154 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.103122    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"]
Jul 28 20:21:04.144574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.144540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjn58\" (UniqueName: \"kubernetes.io/projected/754eaa7a-c4b9-4d59-8cf6-54499a165e75-kube-api-access-tjn58\") pod \"kuadrant-operator-controller-manager-86878c769f-7zd55\" (UID: \"754eaa7a-c4b9-4d59-8cf6-54499a165e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:21:04.144753 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.144683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/754eaa7a-c4b9-4d59-8cf6-54499a165e75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7zd55\" (UID: \"754eaa7a-c4b9-4d59-8cf6-54499a165e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:21:04.245745 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.245708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/754eaa7a-c4b9-4d59-8cf6-54499a165e75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7zd55\" (UID: \"754eaa7a-c4b9-4d59-8cf6-54499a165e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:21:04.245904 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.245789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjn58\" (UniqueName: \"kubernetes.io/projected/754eaa7a-c4b9-4d59-8cf6-54499a165e75-kube-api-access-tjn58\") pod \"kuadrant-operator-controller-manager-86878c769f-7zd55\" (UID: \"754eaa7a-c4b9-4d59-8cf6-54499a165e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:21:04.246174 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.246133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/754eaa7a-c4b9-4d59-8cf6-54499a165e75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7zd55\" (UID: \"754eaa7a-c4b9-4d59-8cf6-54499a165e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:21:04.257504 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.257475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjn58\" (UniqueName: \"kubernetes.io/projected/754eaa7a-c4b9-4d59-8cf6-54499a165e75-kube-api-access-tjn58\") pod \"kuadrant-operator-controller-manager-86878c769f-7zd55\" (UID: \"754eaa7a-c4b9-4d59-8cf6-54499a165e75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:21:04.380390 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.380274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:21:04.524125 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:04.524104    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"]
Jul 28 20:21:04.526045 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:21:04.526020    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754eaa7a_c4b9_4d59_8cf6_54499a165e75.slice/crio-b27fc7fbc44558594388fbbb4cbc82872c56e9fd46f985f9be952bcd922fc3d8 WatchSource:0}: Error finding container b27fc7fbc44558594388fbbb4cbc82872c56e9fd46f985f9be952bcd922fc3d8: Status 404 returned error can't find the container with id b27fc7fbc44558594388fbbb4cbc82872c56e9fd46f985f9be952bcd922fc3d8
Jul 28 20:21:05.404774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:05.404733    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55" event={"ID":"754eaa7a-c4b9-4d59-8cf6-54499a165e75","Type":"ContainerStarted","Data":"a907b15d583613e5880099f2719bf58fed8fb0a32af44ba319afeb803143522c"}
Jul 28 20:21:05.405181 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:05.404797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55" event={"ID":"754eaa7a-c4b9-4d59-8cf6-54499a165e75","Type":"ContainerStarted","Data":"b27fc7fbc44558594388fbbb4cbc82872c56e9fd46f985f9be952bcd922fc3d8"}
Jul 28 20:21:05.405181 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:05.404823    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:21:05.444817 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:05.444773    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55" podStartSLOduration=1.444759141 podStartE2EDuration="1.444759141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:21:05.442691321 +0000 UTC m=+1315.297002386" watchObservedRunningTime="2026-07-28 20:21:05.444759141 +0000 UTC m=+1315.299070206"
Jul 28 20:21:16.410784 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:21:16.410755    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7zd55"
Jul 28 20:23:56.213594 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.213562    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"]
Jul 28 20:23:56.218187 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.218155    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.221642 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.221620    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-73dda84b-kserve-self-signed-certs\""
Jul 28 20:23:56.243563 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.243530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"]
Jul 28 20:23:56.315277 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.315240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-model-cache\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.315469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.315280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-tmp-dir\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.315469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.315363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-home\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.315469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.315394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mnzv\" (UniqueName: \"kubernetes.io/projected/639405cf-6d2b-4301-ae69-7a6053bfa413-kube-api-access-5mnzv\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.315598 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.315465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639405cf-6d2b-4301-ae69-7a6053bfa413-tls-certs\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.315598 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.315497    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-dshm\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.416776 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.416734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-model-cache\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.416776 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.416785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-tmp-dir\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.417023 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.416815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-home\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.417023 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.416843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mnzv\" (UniqueName: \"kubernetes.io/projected/639405cf-6d2b-4301-ae69-7a6053bfa413-kube-api-access-5mnzv\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.417023 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.416879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639405cf-6d2b-4301-ae69-7a6053bfa413-tls-certs\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.417023 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.416899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-dshm\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.417232 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.417215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-model-cache\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.417305 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.417250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-tmp-dir\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.417364 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.417317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-home\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.419210 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.419188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-dshm\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.419358 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.419338    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639405cf-6d2b-4301-ae69-7a6053bfa413-tls-certs\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.426741 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.426715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mnzv\" (UniqueName: \"kubernetes.io/projected/639405cf-6d2b-4301-ae69-7a6053bfa413-kube-api-access-5mnzv\") pod \"e2e-same-id-a-73dda84b-kserve-857894978c-68j2q\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") " pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.531927 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.531888    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:23:56.701278 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:56.701249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"]
Jul 28 20:23:56.703049 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:23:56.703018    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod639405cf_6d2b_4301_ae69_7a6053bfa413.slice/crio-6f9bce20a224a4f5c784dbb9183be73a00063b54ca180e0484bb8f923bf6169b WatchSource:0}: Error finding container 6f9bce20a224a4f5c784dbb9183be73a00063b54ca180e0484bb8f923bf6169b: Status 404 returned error can't find the container with id 6f9bce20a224a4f5c784dbb9183be73a00063b54ca180e0484bb8f923bf6169b
Jul 28 20:23:57.154751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:57.154714    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:23:57.220886 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:57.220848    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q" event={"ID":"639405cf-6d2b-4301-ae69-7a6053bfa413","Type":"ContainerStarted","Data":"9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871"}
Jul 28 20:23:57.221275 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:57.220897    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q" event={"ID":"639405cf-6d2b-4301-ae69-7a6053bfa413","Type":"ContainerStarted","Data":"6f9bce20a224a4f5c784dbb9183be73a00063b54ca180e0484bb8f923bf6169b"}
Jul 28 20:23:57.243575 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:23:57.243528    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q" podStartSLOduration=1.243512699 podStartE2EDuration="1.243512699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:23:57.240659529 +0000 UTC m=+1487.094970596" watchObservedRunningTime="2026-07-28 20:23:57.243512699 +0000 UTC m=+1487.097823764"
Jul 28 20:24:06.532884 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:06.532796    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:24:06.532884 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:06.532878    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:24:06.545496 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:06.545472    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:24:07.282060 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.282032    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:24:07.495555 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.495526    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"]
Jul 28 20:24:07.498957 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.498937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.504541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.504524    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-73dda84b-kserve-self-signed-certs\""
Jul 28 20:24:07.512220 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.512200    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"]
Jul 28 20:24:07.532760 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.532696    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:07.617497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.617462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-home\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.617497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.617499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tls-certs\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.617906 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.617526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-model-cache\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.617906 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.617572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-dshm\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.617906 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.617642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tmp-dir\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.617906 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.617674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfc9t\" (UniqueName: \"kubernetes.io/projected/0cbe0b24-3cb5-4b84-ba46-70947db118d8-kube-api-access-gfc9t\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.718787 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.718752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tmp-dir\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.718950 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.718802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfc9t\" (UniqueName: \"kubernetes.io/projected/0cbe0b24-3cb5-4b84-ba46-70947db118d8-kube-api-access-gfc9t\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.718950 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.718921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-home\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.719032 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.718975    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tls-certs\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.719032 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.719022    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-model-cache\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.719132 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.719066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-dshm\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.719210 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.719191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tmp-dir\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.719315 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.719266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-home\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.719386 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.719365    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-model-cache\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.721818 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.721790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-dshm\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.722005 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.721990    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tls-certs\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.728360 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.728316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfc9t\" (UniqueName: \"kubernetes.io/projected/0cbe0b24-3cb5-4b84-ba46-70947db118d8-kube-api-access-gfc9t\") pod \"e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") " pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.808972 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.808905    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:07.945483 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:07.945455    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"]
Jul 28 20:24:07.947502 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:24:07.947474    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cbe0b24_3cb5_4b84_ba46_70947db118d8.slice/crio-07b64a791bbd7606190ae91ef8a0356d474ed0aa08c06936eb681dff09def885 WatchSource:0}: Error finding container 07b64a791bbd7606190ae91ef8a0356d474ed0aa08c06936eb681dff09def885: Status 404 returned error can't find the container with id 07b64a791bbd7606190ae91ef8a0356d474ed0aa08c06936eb681dff09def885
Jul 28 20:24:08.277336 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:08.277284    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc" event={"ID":"0cbe0b24-3cb5-4b84-ba46-70947db118d8","Type":"ContainerStarted","Data":"e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996"}
Jul 28 20:24:08.277516 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:08.277348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc" event={"ID":"0cbe0b24-3cb5-4b84-ba46-70947db118d8","Type":"ContainerStarted","Data":"07b64a791bbd7606190ae91ef8a0356d474ed0aa08c06936eb681dff09def885"}
Jul 28 20:24:08.297797 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:08.297747    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc" podStartSLOduration=1.297731441 podStartE2EDuration="1.297731441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:24:08.296130942 +0000 UTC m=+1498.150442009" watchObservedRunningTime="2026-07-28 20:24:08.297731441 +0000 UTC m=+1498.152042506"
Jul 28 20:24:17.809934 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:17.809897    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:17.810334 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:17.809957    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:17.822965 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:17.822939    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:18.340984 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:18.340951    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:34.957381 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:34.957280    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"]
Jul 28 20:24:35.006735 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:35.006693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:36.615489 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:36.615459    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"]
Jul 28 20:24:36.672059 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:36.672018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:49.976454 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:49.976392    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q" podUID="639405cf-6d2b-4301-ae69-7a6053bfa413" containerName="main" containerID="cri-o://9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871" gracePeriod=45
Jul 28 20:24:50.226660 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.226599    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:24:50.272192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272159    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-tmp-dir\") pod \"639405cf-6d2b-4301-ae69-7a6053bfa413\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") "
Jul 28 20:24:50.272372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272202    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-home\") pod \"639405cf-6d2b-4301-ae69-7a6053bfa413\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") "
Jul 28 20:24:50.272372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272237    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639405cf-6d2b-4301-ae69-7a6053bfa413-tls-certs\") pod \"639405cf-6d2b-4301-ae69-7a6053bfa413\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") "
Jul 28 20:24:50.272372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272273    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-model-cache\") pod \"639405cf-6d2b-4301-ae69-7a6053bfa413\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") "
Jul 28 20:24:50.272372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272305    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-dshm\") pod \"639405cf-6d2b-4301-ae69-7a6053bfa413\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") "
Jul 28 20:24:50.272372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mnzv\" (UniqueName: \"kubernetes.io/projected/639405cf-6d2b-4301-ae69-7a6053bfa413-kube-api-access-5mnzv\") pod \"639405cf-6d2b-4301-ae69-7a6053bfa413\" (UID: \"639405cf-6d2b-4301-ae69-7a6053bfa413\") "
Jul 28 20:24:50.272664 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272517    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "639405cf-6d2b-4301-ae69-7a6053bfa413" (UID: "639405cf-6d2b-4301-ae69-7a6053bfa413"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:50.272664 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272534    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-home" (OuterVolumeSpecName: "home") pod "639405cf-6d2b-4301-ae69-7a6053bfa413" (UID: "639405cf-6d2b-4301-ae69-7a6053bfa413"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:50.272664 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272643    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-tmp-dir\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:50.272664 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272660    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-home\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:50.272925 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.272903    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-model-cache" (OuterVolumeSpecName: "model-cache") pod "639405cf-6d2b-4301-ae69-7a6053bfa413" (UID: "639405cf-6d2b-4301-ae69-7a6053bfa413"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:50.274512 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.274486    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639405cf-6d2b-4301-ae69-7a6053bfa413-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "639405cf-6d2b-4301-ae69-7a6053bfa413" (UID: "639405cf-6d2b-4301-ae69-7a6053bfa413"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:24:50.274798 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.274774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639405cf-6d2b-4301-ae69-7a6053bfa413-kube-api-access-5mnzv" (OuterVolumeSpecName: "kube-api-access-5mnzv") pod "639405cf-6d2b-4301-ae69-7a6053bfa413" (UID: "639405cf-6d2b-4301-ae69-7a6053bfa413"). InnerVolumeSpecName "kube-api-access-5mnzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:24:50.274798 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.274781    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-dshm" (OuterVolumeSpecName: "dshm") pod "639405cf-6d2b-4301-ae69-7a6053bfa413" (UID: "639405cf-6d2b-4301-ae69-7a6053bfa413"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:50.373533 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.373499    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-model-cache\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:50.373533 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.373526    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639405cf-6d2b-4301-ae69-7a6053bfa413-dshm\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:50.373533 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.373536    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mnzv\" (UniqueName: \"kubernetes.io/projected/639405cf-6d2b-4301-ae69-7a6053bfa413-kube-api-access-5mnzv\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:50.373737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.373547    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639405cf-6d2b-4301-ae69-7a6053bfa413-tls-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:50.485366 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.485336    2582 generic.go:356] "Generic (PLEG): container finished" podID="639405cf-6d2b-4301-ae69-7a6053bfa413" containerID="9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871" exitCode=0
Jul 28 20:24:50.485551 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.485382    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"
Jul 28 20:24:50.485551 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.485428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q" event={"ID":"639405cf-6d2b-4301-ae69-7a6053bfa413","Type":"ContainerDied","Data":"9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871"}
Jul 28 20:24:50.485551 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.485478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q" event={"ID":"639405cf-6d2b-4301-ae69-7a6053bfa413","Type":"ContainerDied","Data":"6f9bce20a224a4f5c784dbb9183be73a00063b54ca180e0484bb8f923bf6169b"}
Jul 28 20:24:50.485551 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.485503    2582 scope.go:117] "RemoveContainer" containerID="9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871"
Jul 28 20:24:50.494643 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.494628    2582 scope.go:117] "RemoveContainer" containerID="9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871"
Jul 28 20:24:50.494903 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:24:50.494886    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871\": container with ID starting with 9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871 not found: ID does not exist" containerID="9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871"
Jul 28 20:24:50.494947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.494911    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871"} err="failed to get container status \"9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871\": rpc error: code = NotFound desc = could not find container \"9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871\": container with ID starting with 9e3d57624ab60b5582993a543a186c9268d75bb6c993c1e8943bcba8688bd871 not found: ID does not exist"
Jul 28 20:24:50.506930 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.506906    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"]
Jul 28 20:24:50.512304 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.512275    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-73dda84b-kserve-857894978c-68j2q"]
Jul 28 20:24:50.582636 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:50.582609    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="639405cf-6d2b-4301-ae69-7a6053bfa413" path="/var/lib/kubelet/pods/639405cf-6d2b-4301-ae69-7a6053bfa413/volumes"
Jul 28 20:24:51.634416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.634373    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc" podUID="0cbe0b24-3cb5-4b84-ba46-70947db118d8" containerName="main" containerID="cri-o://e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996" gracePeriod=45
Jul 28 20:24:51.896366 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.896310    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:51.988975 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.988939    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tmp-dir\") pod \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") "
Jul 28 20:24:51.989152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.988984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tls-certs\") pod \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") "
Jul 28 20:24:51.989152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.989009    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-dshm\") pod \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") "
Jul 28 20:24:51.989152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.989043    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-model-cache\") pod \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") "
Jul 28 20:24:51.989152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.989074    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-home\") pod \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") "
Jul 28 20:24:51.989152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.989109    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfc9t\" (UniqueName: \"kubernetes.io/projected/0cbe0b24-3cb5-4b84-ba46-70947db118d8-kube-api-access-gfc9t\") pod \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\" (UID: \"0cbe0b24-3cb5-4b84-ba46-70947db118d8\") "
Jul 28 20:24:51.989439 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.989249    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cbe0b24-3cb5-4b84-ba46-70947db118d8" (UID: "0cbe0b24-3cb5-4b84-ba46-70947db118d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:51.989439 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.989384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cbe0b24-3cb5-4b84-ba46-70947db118d8" (UID: "0cbe0b24-3cb5-4b84-ba46-70947db118d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:51.989439 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.989408    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-home" (OuterVolumeSpecName: "home") pod "0cbe0b24-3cb5-4b84-ba46-70947db118d8" (UID: "0cbe0b24-3cb5-4b84-ba46-70947db118d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:51.989572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.989451    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tmp-dir\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:51.991101 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.991073    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-dshm" (OuterVolumeSpecName: "dshm") pod "0cbe0b24-3cb5-4b84-ba46-70947db118d8" (UID: "0cbe0b24-3cb5-4b84-ba46-70947db118d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:51.991101 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.991095    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cbe0b24-3cb5-4b84-ba46-70947db118d8" (UID: "0cbe0b24-3cb5-4b84-ba46-70947db118d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:24:51.991236 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:51.991127    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbe0b24-3cb5-4b84-ba46-70947db118d8-kube-api-access-gfc9t" (OuterVolumeSpecName: "kube-api-access-gfc9t") pod "0cbe0b24-3cb5-4b84-ba46-70947db118d8" (UID: "0cbe0b24-3cb5-4b84-ba46-70947db118d8"). InnerVolumeSpecName "kube-api-access-gfc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:24:52.090141 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.090111    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-home\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:52.090141 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.090137    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfc9t\" (UniqueName: \"kubernetes.io/projected/0cbe0b24-3cb5-4b84-ba46-70947db118d8-kube-api-access-gfc9t\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:52.090141 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.090148    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe0b24-3cb5-4b84-ba46-70947db118d8-tls-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:52.090371 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.090156    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-dshm\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:52.090371 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.090164    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe0b24-3cb5-4b84-ba46-70947db118d8-model-cache\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:24:52.496225 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.496188    2582 generic.go:356] "Generic (PLEG): container finished" podID="0cbe0b24-3cb5-4b84-ba46-70947db118d8" containerID="e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996" exitCode=0
Jul 28 20:24:52.496416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.496248    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"
Jul 28 20:24:52.496416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.496274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc" event={"ID":"0cbe0b24-3cb5-4b84-ba46-70947db118d8","Type":"ContainerDied","Data":"e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996"}
Jul 28 20:24:52.496416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.496337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc" event={"ID":"0cbe0b24-3cb5-4b84-ba46-70947db118d8","Type":"ContainerDied","Data":"07b64a791bbd7606190ae91ef8a0356d474ed0aa08c06936eb681dff09def885"}
Jul 28 20:24:52.496416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.496357    2582 scope.go:117] "RemoveContainer" containerID="e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996"
Jul 28 20:24:52.505598 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.505582    2582 scope.go:117] "RemoveContainer" containerID="e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996"
Jul 28 20:24:52.505853 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:24:52.505833    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996\": container with ID starting with e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996 not found: ID does not exist" containerID="e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996"
Jul 28 20:24:52.505924 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.505865    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996"} err="failed to get container status \"e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996\": rpc error: code = NotFound desc = could not find container \"e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996\": container with ID starting with e69b828da777162ec2cdf26f93a010ce859466cb38f5e2bda86ddec1fb7ba996 not found: ID does not exist"
Jul 28 20:24:52.517618 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.517594    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"]
Jul 28 20:24:52.521433 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.521413    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-73dda84b-kserve-7df5984b95-dvjjc"]
Jul 28 20:24:52.582391 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:24:52.582368    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cbe0b24-3cb5-4b84-ba46-70947db118d8" path="/var/lib/kubelet/pods/0cbe0b24-3cb5-4b84-ba46-70947db118d8/volumes"
Jul 28 20:25:31.905447 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:25:31.905410    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:26:34.274630 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:26:34.274592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:27:58.793708 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:27:58.793628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:05.563469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:05.563426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:05.679199 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:05.679164    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:20.518494 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:20.518459    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:23.574364 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:23.574327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:23.886003 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:23.885908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:24.093376 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:24.093335    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:30.461655 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:30.461610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:30.630102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:30.630065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:30.731546 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:30.731452    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:31.050166 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:31.050122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:31.760146 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:31.760107    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:33.706651 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.706616    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:33.824055 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.824018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:33.922392 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.922358    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"]
Jul 28 20:28:33.922775 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.922762    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbe0b24-3cb5-4b84-ba46-70947db118d8" containerName="main"
Jul 28 20:28:33.922824 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.922777    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbe0b24-3cb5-4b84-ba46-70947db118d8" containerName="main"
Jul 28 20:28:33.922824 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.922794    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639405cf-6d2b-4301-ae69-7a6053bfa413" containerName="main"
Jul 28 20:28:33.922824 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.922800    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="639405cf-6d2b-4301-ae69-7a6053bfa413" containerName="main"
Jul 28 20:28:33.922932 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.922857    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="639405cf-6d2b-4301-ae69-7a6053bfa413" containerName="main"
Jul 28 20:28:33.922932 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.922871    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbe0b24-3cb5-4b84-ba46-70947db118d8" containerName="main"
Jul 28 20:28:33.925525 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.925506    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:33.930307 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.930273    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a4f1a2e9-serving-cert\""
Jul 28 20:28:33.965123 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:33.965037    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"]
Jul 28 20:28:34.002824 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.002570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:34.025541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.025507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:34.025541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.025551    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ft5p\" (UniqueName: \"kubernetes.io/projected/cf436b97-f292-4179-86ea-f63fa0a8d360-kube-api-access-8ft5p\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:34.025775 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.025691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf436b97-f292-4179-86ea-f63fa0a8d360-openshift-service-ca\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:34.126642 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.126608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf436b97-f292-4179-86ea-f63fa0a8d360-openshift-service-ca\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:34.126847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.126693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:34.126847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.126727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ft5p\" (UniqueName: \"kubernetes.io/projected/cf436b97-f292-4179-86ea-f63fa0a8d360-kube-api-access-8ft5p\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:34.126974 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:28:34.126853    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-serving-cert: secret "maas-api-e2e-ait-a4f1a2e9-serving-cert" not found
Jul 28 20:28:34.126974 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:28:34.126930    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls podName:cf436b97-f292-4179-86ea-f63fa0a8d360 nodeName:}" failed. No retries permitted until 2026-07-28 20:28:34.626911202 +0000 UTC m=+1764.481222248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls") pod "maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx" (UID: "cf436b97-f292-4179-86ea-f63fa0a8d360") : secret "maas-api-e2e-ait-a4f1a2e9-serving-cert" not found
Jul 28 20:28:34.127451 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.127429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf436b97-f292-4179-86ea-f63fa0a8d360-openshift-service-ca\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:34.153091 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.153055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ft5p\" (UniqueName: \"kubernetes.io/projected/cf436b97-f292-4179-86ea-f63fa0a8d360-kube-api-access-8ft5p\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:34.631715 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.631677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:34.631897 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:28:34.631831    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-serving-cert: secret "maas-api-e2e-ait-a4f1a2e9-serving-cert" not found
Jul 28 20:28:34.631947 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:28:34.631907    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls podName:cf436b97-f292-4179-86ea-f63fa0a8d360 nodeName:}" failed. No retries permitted until 2026-07-28 20:28:35.631886576 +0000 UTC m=+1765.486197622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls") pod "maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx" (UID: "cf436b97-f292-4179-86ea-f63fa0a8d360") : secret "maas-api-e2e-ait-a4f1a2e9-serving-cert" not found
Jul 28 20:28:34.726832 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.726789    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:34.830905 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.830865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:34.936207 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.936123    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"]
Jul 28 20:28:34.939940 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.939924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:28:34.945540 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.945516    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a4f1a2e9\""
Jul 28 20:28:34.945670 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.945517    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a4f1a2e9-dockercfg-5fgqt\""
Jul 28 20:28:34.962075 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.962044    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"]
Jul 28 20:28:34.999449 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:34.999420    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"]
Jul 28 20:28:35.003381 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.003356    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:35.018733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.018706    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"]
Jul 28 20:28:35.035013 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.034982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f212ae96-cec6-4589-9f75-eb63f762cde8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl\" (UID: \"f212ae96-cec6-4589-9f75-eb63f762cde8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:28:35.035216 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.035020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwwq\" (UniqueName: \"kubernetes.io/projected/f212ae96-cec6-4589-9f75-eb63f762cde8-kube-api-access-wmwwq\") pod \"payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl\" (UID: \"f212ae96-cec6-4589-9f75-eb63f762cde8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:28:35.136211 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.136176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f212ae96-cec6-4589-9f75-eb63f762cde8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl\" (UID: \"f212ae96-cec6-4589-9f75-eb63f762cde8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:28:35.136211 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.136216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a9fe96-c950-4d53-b2be-3c7b87e98dac-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl\" (UID: \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:35.136437 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.136239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwwq\" (UniqueName: \"kubernetes.io/projected/f212ae96-cec6-4589-9f75-eb63f762cde8-kube-api-access-wmwwq\") pod \"payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl\" (UID: \"f212ae96-cec6-4589-9f75-eb63f762cde8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:28:35.136437 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.136255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5bs\" (UniqueName: \"kubernetes.io/projected/18a9fe96-c950-4d53-b2be-3c7b87e98dac-kube-api-access-qn5bs\") pod \"payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl\" (UID: \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:35.136869 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.136848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f212ae96-cec6-4589-9f75-eb63f762cde8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl\" (UID: \"f212ae96-cec6-4589-9f75-eb63f762cde8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:28:35.146285 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.146260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwwq\" (UniqueName: \"kubernetes.io/projected/f212ae96-cec6-4589-9f75-eb63f762cde8-kube-api-access-wmwwq\") pod \"payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl\" (UID: \"f212ae96-cec6-4589-9f75-eb63f762cde8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:28:35.237818 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.237781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a9fe96-c950-4d53-b2be-3c7b87e98dac-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl\" (UID: \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:35.237999 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.237830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5bs\" (UniqueName: \"kubernetes.io/projected/18a9fe96-c950-4d53-b2be-3c7b87e98dac-kube-api-access-qn5bs\") pod \"payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl\" (UID: \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:35.238595 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.238561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a9fe96-c950-4d53-b2be-3c7b87e98dac-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl\" (UID: \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:35.247847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.247819    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5bs\" (UniqueName: \"kubernetes.io/projected/18a9fe96-c950-4d53-b2be-3c7b87e98dac-kube-api-access-qn5bs\") pod \"payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl\" (UID: \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:35.250701 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.250676    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:28:35.316124 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.315949    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:35.420647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.418090    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"]
Jul 28 20:28:35.503122 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.503092    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"]
Jul 28 20:28:35.507143 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:28:35.507116    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a9fe96_c950_4d53_b2be_3c7b87e98dac.slice/crio-0f4e7d78b76a8cf226834ccc3a78f9a9ce9f168d87b4136ddd6e5a89c9b70ca8 WatchSource:0}: Error finding container 0f4e7d78b76a8cf226834ccc3a78f9a9ce9f168d87b4136ddd6e5a89c9b70ca8: Status 404 returned error can't find the container with id 0f4e7d78b76a8cf226834ccc3a78f9a9ce9f168d87b4136ddd6e5a89c9b70ca8
Jul 28 20:28:35.581735 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.581698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl" event={"ID":"f212ae96-cec6-4589-9f75-eb63f762cde8","Type":"ContainerStarted","Data":"14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b"}
Jul 28 20:28:35.581932 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.581758    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl" event={"ID":"f212ae96-cec6-4589-9f75-eb63f762cde8","Type":"ContainerStarted","Data":"d712943cdc1f3228656dd4e5d03b835c0fe7685d9212f8bea444980bed980370"}
Jul 28 20:28:35.581932 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.581892    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:28:35.583253 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.583227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl" event={"ID":"18a9fe96-c950-4d53-b2be-3c7b87e98dac","Type":"ContainerStarted","Data":"be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7"}
Jul 28 20:28:35.583383 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.583286    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:35.583383 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.583326    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl" event={"ID":"18a9fe96-c950-4d53-b2be-3c7b87e98dac","Type":"ContainerStarted","Data":"0f4e7d78b76a8cf226834ccc3a78f9a9ce9f168d87b4136ddd6e5a89c9b70ca8"}
Jul 28 20:28:35.619867 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.619815    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl" podStartSLOduration=1.619797579 podStartE2EDuration="1.619797579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:28:35.61780872 +0000 UTC m=+1765.472119787" watchObservedRunningTime="2026-07-28 20:28:35.619797579 +0000 UTC m=+1765.474108652"
Jul 28 20:28:35.642032 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.641994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:35.644676 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.644644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls\") pod \"maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:35.646124 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.646081    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl" podStartSLOduration=1.646067444 podStartE2EDuration="1.646067444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:28:35.645163907 +0000 UTC m=+1765.499474995" watchObservedRunningTime="2026-07-28 20:28:35.646067444 +0000 UTC m=+1765.500378509"
Jul 28 20:28:35.736035 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.735996    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:35.890432 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.890351    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"]
Jul 28 20:28:35.893001 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:28:35.892974    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf436b97_f292_4179_86ea_f63fa0a8d360.slice/crio-6b260eeef14d38e895b96d09720a067af1d9ad1560aed5b6325ac72deeb82548 WatchSource:0}: Error finding container 6b260eeef14d38e895b96d09720a067af1d9ad1560aed5b6325ac72deeb82548: Status 404 returned error can't find the container with id 6b260eeef14d38e895b96d09720a067af1d9ad1560aed5b6325ac72deeb82548
Jul 28 20:28:35.894499 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:35.894483    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:28:36.588737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:36.588700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx" event={"ID":"cf436b97-f292-4179-86ea-f63fa0a8d360","Type":"ContainerStarted","Data":"cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a"}
Jul 28 20:28:36.588952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:36.588755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx" event={"ID":"cf436b97-f292-4179-86ea-f63fa0a8d360","Type":"ContainerStarted","Data":"6b260eeef14d38e895b96d09720a067af1d9ad1560aed5b6325ac72deeb82548"}
Jul 28 20:28:36.588952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:36.588817    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:36.643866 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:36.643813    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx" podStartSLOduration=3.091479608 podStartE2EDuration="3.643795076s" totalImagesPullingTime="552.315468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:28:36.642936644 +0000 UTC m=+1766.497247711" watchObservedRunningTime="2026-07-28 20:28:36.643795076 +0000 UTC m=+1766.498106165"
Jul 28 20:28:42.599346 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:42.599314    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:28:46.590508 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:46.590457    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl" podUID="f212ae96-cec6-4589-9f75-eb63f762cde8" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.54:9004: connect: connection refused"
Jul 28 20:28:46.590959 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:46.590762    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:28:55.251789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:55.251742    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl" podUID="f212ae96-cec6-4589-9f75-eb63f762cde8" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.54:9004: connect: connection refused"
Jul 28 20:28:56.590282 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:28:56.590255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:29:04.771790 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:04.771752    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"]
Jul 28 20:29:04.772660 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:04.772620    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx" podUID="cf436b97-f292-4179-86ea-f63fa0a8d360" containerName="maas-api" containerID="cri-o://cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a" gracePeriod=30
Jul 28 20:29:04.819467 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:04.819429    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:04.863183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:04.863149    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"]
Jul 28 20:29:04.863516 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:04.863486    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl" podUID="18a9fe96-c950-4d53-b2be-3c7b87e98dac" containerName="payload-processing" containerID="cri-o://be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7" gracePeriod=30
Jul 28 20:29:04.876390 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:04.876364    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"]
Jul 28 20:29:04.876570 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:04.876550    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl" podUID="f212ae96-cec6-4589-9f75-eb63f762cde8" containerName="payload-pre-processing" containerID="cri-o://14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b" gracePeriod=30
Jul 28 20:29:04.943544 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:04.943511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:05.043993 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.043419    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:05.140941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.140908    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:29:05.174225 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.174200    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:29:05.182641 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.182616    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:29:05.187059 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187040    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls\") pod \"cf436b97-f292-4179-86ea-f63fa0a8d360\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") "
Jul 28 20:29:05.187144 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ft5p\" (UniqueName: \"kubernetes.io/projected/cf436b97-f292-4179-86ea-f63fa0a8d360-kube-api-access-8ft5p\") pod \"cf436b97-f292-4179-86ea-f63fa0a8d360\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") "
Jul 28 20:29:05.187144 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f212ae96-cec6-4589-9f75-eb63f762cde8-plugins-config-volume\") pod \"f212ae96-cec6-4589-9f75-eb63f762cde8\" (UID: \"f212ae96-cec6-4589-9f75-eb63f762cde8\") "
Jul 28 20:29:05.187238 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187168    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf436b97-f292-4179-86ea-f63fa0a8d360-openshift-service-ca\") pod \"cf436b97-f292-4179-86ea-f63fa0a8d360\" (UID: \"cf436b97-f292-4179-86ea-f63fa0a8d360\") "
Jul 28 20:29:05.187238 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187197    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn5bs\" (UniqueName: \"kubernetes.io/projected/18a9fe96-c950-4d53-b2be-3c7b87e98dac-kube-api-access-qn5bs\") pod \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\" (UID: \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\") "
Jul 28 20:29:05.187360 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187244    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a9fe96-c950-4d53-b2be-3c7b87e98dac-plugins-config-volume\") pod \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\" (UID: \"18a9fe96-c950-4d53-b2be-3c7b87e98dac\") "
Jul 28 20:29:05.187360 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187275    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmwwq\" (UniqueName: \"kubernetes.io/projected/f212ae96-cec6-4589-9f75-eb63f762cde8-kube-api-access-wmwwq\") pod \"f212ae96-cec6-4589-9f75-eb63f762cde8\" (UID: \"f212ae96-cec6-4589-9f75-eb63f762cde8\") "
Jul 28 20:29:05.187601 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187559    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf436b97-f292-4179-86ea-f63fa0a8d360-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf436b97-f292-4179-86ea-f63fa0a8d360" (UID: "cf436b97-f292-4179-86ea-f63fa0a8d360"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:29:05.187601 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187557    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f212ae96-cec6-4589-9f75-eb63f762cde8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f212ae96-cec6-4589-9f75-eb63f762cde8" (UID: "f212ae96-cec6-4589-9f75-eb63f762cde8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:29:05.187791 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.187697    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a9fe96-c950-4d53-b2be-3c7b87e98dac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18a9fe96-c950-4d53-b2be-3c7b87e98dac" (UID: "18a9fe96-c950-4d53-b2be-3c7b87e98dac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:29:05.189480 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.189457    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf436b97-f292-4179-86ea-f63fa0a8d360" (UID: "cf436b97-f292-4179-86ea-f63fa0a8d360"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:29:05.189670 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.189652    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf436b97-f292-4179-86ea-f63fa0a8d360-kube-api-access-8ft5p" (OuterVolumeSpecName: "kube-api-access-8ft5p") pod "cf436b97-f292-4179-86ea-f63fa0a8d360" (UID: "cf436b97-f292-4179-86ea-f63fa0a8d360"). InnerVolumeSpecName "kube-api-access-8ft5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:29:05.189732 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.189699    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a9fe96-c950-4d53-b2be-3c7b87e98dac-kube-api-access-qn5bs" (OuterVolumeSpecName: "kube-api-access-qn5bs") pod "18a9fe96-c950-4d53-b2be-3c7b87e98dac" (UID: "18a9fe96-c950-4d53-b2be-3c7b87e98dac"). InnerVolumeSpecName "kube-api-access-qn5bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:29:05.189770 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.189723    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f212ae96-cec6-4589-9f75-eb63f762cde8-kube-api-access-wmwwq" (OuterVolumeSpecName: "kube-api-access-wmwwq") pod "f212ae96-cec6-4589-9f75-eb63f762cde8" (UID: "f212ae96-cec6-4589-9f75-eb63f762cde8"). InnerVolumeSpecName "kube-api-access-wmwwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:29:05.204574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.204528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:05.289350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.288671    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a9fe96-c950-4d53-b2be-3c7b87e98dac-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:05.289350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.288755    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwwq\" (UniqueName: \"kubernetes.io/projected/f212ae96-cec6-4589-9f75-eb63f762cde8-kube-api-access-wmwwq\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:05.289350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.288796    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf436b97-f292-4179-86ea-f63fa0a8d360-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:05.289350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.288813    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ft5p\" (UniqueName: \"kubernetes.io/projected/cf436b97-f292-4179-86ea-f63fa0a8d360-kube-api-access-8ft5p\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:05.289350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.288827    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f212ae96-cec6-4589-9f75-eb63f762cde8-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:05.289350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.288852    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf436b97-f292-4179-86ea-f63fa0a8d360-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:05.289350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.288866    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn5bs\" (UniqueName: \"kubernetes.io/projected/18a9fe96-c950-4d53-b2be-3c7b87e98dac-kube-api-access-qn5bs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:05.341015 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.340916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:05.729603 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.729511    2582 generic.go:356] "Generic (PLEG): container finished" podID="f212ae96-cec6-4589-9f75-eb63f762cde8" containerID="14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b" exitCode=0
Jul 28 20:29:05.729603 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.729585    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"
Jul 28 20:29:05.729823 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.729596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl" event={"ID":"f212ae96-cec6-4589-9f75-eb63f762cde8","Type":"ContainerDied","Data":"14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b"}
Jul 28 20:29:05.729823 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.729648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl" event={"ID":"f212ae96-cec6-4589-9f75-eb63f762cde8","Type":"ContainerDied","Data":"d712943cdc1f3228656dd4e5d03b835c0fe7685d9212f8bea444980bed980370"}
Jul 28 20:29:05.729823 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.729671    2582 scope.go:117] "RemoveContainer" containerID="14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b"
Jul 28 20:29:05.731073 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.731049    2582 generic.go:356] "Generic (PLEG): container finished" podID="18a9fe96-c950-4d53-b2be-3c7b87e98dac" containerID="be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7" exitCode=0
Jul 28 20:29:05.731189 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.731127    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"
Jul 28 20:29:05.731189 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.731132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl" event={"ID":"18a9fe96-c950-4d53-b2be-3c7b87e98dac","Type":"ContainerDied","Data":"be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7"}
Jul 28 20:29:05.731336 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.731192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl" event={"ID":"18a9fe96-c950-4d53-b2be-3c7b87e98dac","Type":"ContainerDied","Data":"0f4e7d78b76a8cf226834ccc3a78f9a9ce9f168d87b4136ddd6e5a89c9b70ca8"}
Jul 28 20:29:05.732663 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.732635    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf436b97-f292-4179-86ea-f63fa0a8d360" containerID="cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a" exitCode=0
Jul 28 20:29:05.732803 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.732681    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"
Jul 28 20:29:05.732803 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.732702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx" event={"ID":"cf436b97-f292-4179-86ea-f63fa0a8d360","Type":"ContainerDied","Data":"cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a"}
Jul 28 20:29:05.732803 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.732743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx" event={"ID":"cf436b97-f292-4179-86ea-f63fa0a8d360","Type":"ContainerDied","Data":"6b260eeef14d38e895b96d09720a067af1d9ad1560aed5b6325ac72deeb82548"}
Jul 28 20:29:05.741033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.741011    2582 scope.go:117] "RemoveContainer" containerID="14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b"
Jul 28 20:29:05.741883 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:05.741856    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b\": container with ID starting with 14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b not found: ID does not exist" containerID="14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b"
Jul 28 20:29:05.741992 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.741897    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b"} err="failed to get container status \"14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b\": rpc error: code = NotFound desc = could not find container \"14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b\": container with ID starting with 14a82736d3d4d64e077733710dec26e07cde36b19470b208705d317ba109740b not found: ID does not exist"
Jul 28 20:29:05.741992 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.741921    2582 scope.go:117] "RemoveContainer" containerID="be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7"
Jul 28 20:29:05.752558 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.752532    2582 scope.go:117] "RemoveContainer" containerID="be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7"
Jul 28 20:29:05.752842 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:05.752822    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7\": container with ID starting with be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7 not found: ID does not exist" containerID="be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7"
Jul 28 20:29:05.752907 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.752848    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7"} err="failed to get container status \"be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7\": rpc error: code = NotFound desc = could not find container \"be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7\": container with ID starting with be04046fbe57d32408ac58e01a5f475ab77f27feba657b37d64250fc31755fd7 not found: ID does not exist"
Jul 28 20:29:05.752907 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.752864    2582 scope.go:117] "RemoveContainer" containerID="cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a"
Jul 28 20:29:05.760959 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.760932    2582 scope.go:117] "RemoveContainer" containerID="cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a"
Jul 28 20:29:05.761530 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:05.761498    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a\": container with ID starting with cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a not found: ID does not exist" containerID="cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a"
Jul 28 20:29:05.761658 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.761625    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a"} err="failed to get container status \"cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a\": rpc error: code = NotFound desc = could not find container \"cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a\": container with ID starting with cd3fa534fa553a6b5001e3b41911e77e7a425edc38d5bac2a52f4b5fc12ed84a not found: ID does not exist"
Jul 28 20:29:05.767817 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.767793    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"]
Jul 28 20:29:05.774527 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.774503    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1a2e9-747677c8d-z9ncl"]
Jul 28 20:29:05.790737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.790711    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"]
Jul 28 20:29:05.798403 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.798382    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f1a2e9-5c97b88cd8-r6fqx"]
Jul 28 20:29:05.822820 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.822776    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"]
Jul 28 20:29:05.845778 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:05.845736    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1a2e9-5f5d4ff854-8xwvl"]
Jul 28 20:29:06.271000 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.270963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:06.273873 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.273850    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"]
Jul 28 20:29:06.274208 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.274197    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a9fe96-c950-4d53-b2be-3c7b87e98dac" containerName="payload-processing"
Jul 28 20:29:06.274254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.274210    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a9fe96-c950-4d53-b2be-3c7b87e98dac" containerName="payload-processing"
Jul 28 20:29:06.274254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.274226    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f212ae96-cec6-4589-9f75-eb63f762cde8" containerName="payload-pre-processing"
Jul 28 20:29:06.274254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.274232    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f212ae96-cec6-4589-9f75-eb63f762cde8" containerName="payload-pre-processing"
Jul 28 20:29:06.274254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.274242    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf436b97-f292-4179-86ea-f63fa0a8d360" containerName="maas-api"
Jul 28 20:29:06.274254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.274248    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf436b97-f292-4179-86ea-f63fa0a8d360" containerName="maas-api"
Jul 28 20:29:06.274419 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.274324    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a9fe96-c950-4d53-b2be-3c7b87e98dac" containerName="payload-processing"
Jul 28 20:29:06.274419 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.274332    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf436b97-f292-4179-86ea-f63fa0a8d360" containerName="maas-api"
Jul 28 20:29:06.274419 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.274338    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f212ae96-cec6-4589-9f75-eb63f762cde8" containerName="payload-pre-processing"
Jul 28 20:29:06.278726 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.278709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.282429 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.282409    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-addc0a70-gw-openshift-default-dockercfg-6jxg8\""
Jul 28 20:29:06.293092 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.293066    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"]
Jul 28 20:29:06.296614 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.296585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-data\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.296812 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.296793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f08c12-606b-4c6f-8f80-7b305d36799a-istiod-ca-cert\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.296988 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.296972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-socket\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.297163 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.297149    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-certs\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.297321 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.297305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-podinfo\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.297391 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.297375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-envoy\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.297447 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.297415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-credential-socket\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.297512 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.297445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-token\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.297512 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.297478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vnv\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-kube-api-access-c7vnv\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.372708 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.372668    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:06.397974 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.397945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-socket\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-certs\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-podinfo\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-envoy\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-credential-socket\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-token\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vnv\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-kube-api-access-c7vnv\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-socket\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-data\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f08c12-606b-4c6f-8f80-7b305d36799a-istiod-ca-cert\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-credential-socket\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.398702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-data\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.399620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.398780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-certs\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.399620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.399269    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f08c12-606b-4c6f-8f80-7b305d36799a-istiod-ca-cert\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.401421 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.401393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-envoy\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.401995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.401975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-podinfo\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.410743 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.410718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-token\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.411113 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.411092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vnv\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-kube-api-access-c7vnv\") pod \"e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") " pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.584261 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.584182    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a9fe96-c950-4d53-b2be-3c7b87e98dac" path="/var/lib/kubelet/pods/18a9fe96-c950-4d53-b2be-3c7b87e98dac/volumes"
Jul 28 20:29:06.584642 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.584626    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf436b97-f292-4179-86ea-f63fa0a8d360" path="/var/lib/kubelet/pods/cf436b97-f292-4179-86ea-f63fa0a8d360/volumes"
Jul 28 20:29:06.584939 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.584927    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f212ae96-cec6-4589-9f75-eb63f762cde8" path="/var/lib/kubelet/pods/f212ae96-cec6-4589-9f75-eb63f762cde8/volumes"
Jul 28 20:29:06.588759 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.588736    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:06.620376 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.620334    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:06.745314 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.745175    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"]
Jul 28 20:29:06.747926 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:29:06.747883    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f08c12_606b_4c6f_8f80_7b305d36799a.slice/crio-5c631c39b042624c5199cdf67c80134c502fe07eea920245ffcf071956a3c1f0 WatchSource:0}: Error finding container 5c631c39b042624c5199cdf67c80134c502fe07eea920245ffcf071956a3c1f0: Status 404 returned error can't find the container with id 5c631c39b042624c5199cdf67c80134c502fe07eea920245ffcf071956a3c1f0
Jul 28 20:29:06.749878 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.749848    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:29:06.749937 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.749913    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:29:06.749976 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:06.749940    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:29:07.749167 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:07.749123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" event={"ID":"67f08c12-606b-4c6f-8f80-7b305d36799a","Type":"ContainerStarted","Data":"8392be257192b34a7d8a7508e8025e2b3e11de3df95f1a57954382c7da551b95"}
Jul 28 20:29:07.749590 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:07.749197    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" event={"ID":"67f08c12-606b-4c6f-8f80-7b305d36799a","Type":"ContainerStarted","Data":"5c631c39b042624c5199cdf67c80134c502fe07eea920245ffcf071956a3c1f0"}
Jul 28 20:29:07.774588 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:07.774534    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" podStartSLOduration=1.774515415 podStartE2EDuration="1.774515415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:07.773836057 +0000 UTC m=+1797.628147123" watchObservedRunningTime="2026-07-28 20:29:07.774515415 +0000 UTC m=+1797.628826482"
Jul 28 20:29:08.589344 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:08.589309    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:08.594405 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:08.594381    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:08.755313 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:08.755262    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:08.756412 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:08.756392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:09.248642 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.248602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:09.321484 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.321433    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"]
Jul 28 20:29:09.325811 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.325629    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:09.328697 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.328674    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-addc0a70-serving-cert\""
Jul 28 20:29:09.339536 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.339502    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"]
Jul 28 20:29:09.371349 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.371278    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:09.425979 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.425941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9p6w\" (UniqueName: \"kubernetes.io/projected/c589ab26-04bd-408c-a036-5ea42f15fb1c-kube-api-access-p9p6w\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:09.426195 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.425993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c589ab26-04bd-408c-a036-5ea42f15fb1c-openshift-service-ca\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:09.426195 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.426106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c589ab26-04bd-408c-a036-5ea42f15fb1c-maas-api-tls\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:09.527008 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.526903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c589ab26-04bd-408c-a036-5ea42f15fb1c-openshift-service-ca\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:09.527008 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.526997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c589ab26-04bd-408c-a036-5ea42f15fb1c-maas-api-tls\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:09.527234 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.527042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9p6w\" (UniqueName: \"kubernetes.io/projected/c589ab26-04bd-408c-a036-5ea42f15fb1c-kube-api-access-p9p6w\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:09.527234 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:09.527147    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-serving-cert: secret "maas-api-e2e-migrate-addc0a70-serving-cert" not found
Jul 28 20:29:09.527234 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:09.527219    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c589ab26-04bd-408c-a036-5ea42f15fb1c-maas-api-tls podName:c589ab26-04bd-408c-a036-5ea42f15fb1c nodeName:}" failed. No retries permitted until 2026-07-28 20:29:10.027198309 +0000 UTC m=+1799.881509367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c589ab26-04bd-408c-a036-5ea42f15fb1c-maas-api-tls") pod "maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf" (UID: "c589ab26-04bd-408c-a036-5ea42f15fb1c") : secret "maas-api-e2e-migrate-addc0a70-serving-cert" not found
Jul 28 20:29:09.527788 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.527764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c589ab26-04bd-408c-a036-5ea42f15fb1c-openshift-service-ca\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:09.547030 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:09.546998    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9p6w\" (UniqueName: \"kubernetes.io/projected/c589ab26-04bd-408c-a036-5ea42f15fb1c-kube-api-access-p9p6w\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:10.032277 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.032227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c589ab26-04bd-408c-a036-5ea42f15fb1c-maas-api-tls\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:10.035018 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.034983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c589ab26-04bd-408c-a036-5ea42f15fb1c-maas-api-tls\") pod \"maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:10.206432 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.206396    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"]
Jul 28 20:29:10.210089 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.210070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:10.212756 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.212735    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-addc0a70\""
Jul 28 20:29:10.212864 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.212737    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-addc0a70-dockercfg-xtb7j\""
Jul 28 20:29:10.222031 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.222006    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"]
Jul 28 20:29:10.234790 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.234754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352ce7d2-8323-4110-90d0-d6257494cb93-plugins-config-volume\") pod \"payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7\" (UID: \"352ce7d2-8323-4110-90d0-d6257494cb93\") " pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:10.234974 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.234812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb5vf\" (UniqueName: \"kubernetes.io/projected/352ce7d2-8323-4110-90d0-d6257494cb93-kube-api-access-pb5vf\") pod \"payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7\" (UID: \"352ce7d2-8323-4110-90d0-d6257494cb93\") " pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:10.240712 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.240678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:10.265859 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.265822    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"]
Jul 28 20:29:10.270117 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.270095    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:10.278753 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.278689    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"]
Jul 28 20:29:10.336444 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.336336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352ce7d2-8323-4110-90d0-d6257494cb93-plugins-config-volume\") pod \"payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7\" (UID: \"352ce7d2-8323-4110-90d0-d6257494cb93\") " pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:10.336444 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.336425    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73eb9621-6497-4009-bbdc-acfc29f42d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875\" (UID: \"73eb9621-6497-4009-bbdc-acfc29f42d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:10.336737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.336466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb5vf\" (UniqueName: \"kubernetes.io/projected/352ce7d2-8323-4110-90d0-d6257494cb93-kube-api-access-pb5vf\") pod \"payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7\" (UID: \"352ce7d2-8323-4110-90d0-d6257494cb93\") " pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:10.336737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.336542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6x4\" (UniqueName: \"kubernetes.io/projected/73eb9621-6497-4009-bbdc-acfc29f42d9d-kube-api-access-vm6x4\") pod \"payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875\" (UID: \"73eb9621-6497-4009-bbdc-acfc29f42d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:10.337191 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.337164    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352ce7d2-8323-4110-90d0-d6257494cb93-plugins-config-volume\") pod \"payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7\" (UID: \"352ce7d2-8323-4110-90d0-d6257494cb93\") " pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:10.346203 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.346169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb5vf\" (UniqueName: \"kubernetes.io/projected/352ce7d2-8323-4110-90d0-d6257494cb93-kube-api-access-pb5vf\") pod \"payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7\" (UID: \"352ce7d2-8323-4110-90d0-d6257494cb93\") " pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:10.419259 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.419231    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"]
Jul 28 20:29:10.420479 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:29:10.420448    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc589ab26_04bd_408c_a036_5ea42f15fb1c.slice/crio-533d3bcbee332f58647dbaab33c3bb38d379018109040e331ac1170e0ab06aa3 WatchSource:0}: Error finding container 533d3bcbee332f58647dbaab33c3bb38d379018109040e331ac1170e0ab06aa3: Status 404 returned error can't find the container with id 533d3bcbee332f58647dbaab33c3bb38d379018109040e331ac1170e0ab06aa3
Jul 28 20:29:10.437221 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.437189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6x4\" (UniqueName: \"kubernetes.io/projected/73eb9621-6497-4009-bbdc-acfc29f42d9d-kube-api-access-vm6x4\") pod \"payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875\" (UID: \"73eb9621-6497-4009-bbdc-acfc29f42d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:10.437414 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.437327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73eb9621-6497-4009-bbdc-acfc29f42d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875\" (UID: \"73eb9621-6497-4009-bbdc-acfc29f42d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:10.438230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.438200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73eb9621-6497-4009-bbdc-acfc29f42d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875\" (UID: \"73eb9621-6497-4009-bbdc-acfc29f42d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:10.446629 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.446601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6x4\" (UniqueName: \"kubernetes.io/projected/73eb9621-6497-4009-bbdc-acfc29f42d9d-kube-api-access-vm6x4\") pod \"payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875\" (UID: \"73eb9621-6497-4009-bbdc-acfc29f42d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:10.524484 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.524452    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-addc0a70-dockercfg-xtb7j\""
Jul 28 20:29:10.531793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.531756    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:10.553856 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.553812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:10.584686 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.584180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:10.659663 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.659622    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:10.701795 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.701768    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"]
Jul 28 20:29:10.704178 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:29:10.703947    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352ce7d2_8323_4110_90d0_d6257494cb93.slice/crio-1b8b5246926e552ec2461debd6b0151e27d683df2be3105f25c7780e9a9ef1fa WatchSource:0}: Error finding container 1b8b5246926e552ec2461debd6b0151e27d683df2be3105f25c7780e9a9ef1fa: Status 404 returned error can't find the container with id 1b8b5246926e552ec2461debd6b0151e27d683df2be3105f25c7780e9a9ef1fa
Jul 28 20:29:10.767685 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.767430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7" event={"ID":"352ce7d2-8323-4110-90d0-d6257494cb93","Type":"ContainerStarted","Data":"1b8b5246926e552ec2461debd6b0151e27d683df2be3105f25c7780e9a9ef1fa"}
Jul 28 20:29:10.769363 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.769267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf" event={"ID":"c589ab26-04bd-408c-a036-5ea42f15fb1c","Type":"ContainerStarted","Data":"533d3bcbee332f58647dbaab33c3bb38d379018109040e331ac1170e0ab06aa3"}
Jul 28 20:29:10.770331 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:10.770300    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"]
Jul 28 20:29:10.772779 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:29:10.772756    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73eb9621_6497_4009_bbdc_acfc29f42d9d.slice/crio-15f861f91d654acd13488b76e7460279c633ee345831cd76bb0465f46662e913 WatchSource:0}: Error finding container 15f861f91d654acd13488b76e7460279c633ee345831cd76bb0465f46662e913: Status 404 returned error can't find the container with id 15f861f91d654acd13488b76e7460279c633ee345831cd76bb0465f46662e913
Jul 28 20:29:11.775955 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.775916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7" event={"ID":"352ce7d2-8323-4110-90d0-d6257494cb93","Type":"ContainerStarted","Data":"da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3"}
Jul 28 20:29:11.776488 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.776038    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:11.777439 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.777419    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf" event={"ID":"c589ab26-04bd-408c-a036-5ea42f15fb1c","Type":"ContainerStarted","Data":"6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c"}
Jul 28 20:29:11.777550 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.777471    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:11.778832 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.778814    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875" event={"ID":"73eb9621-6497-4009-bbdc-acfc29f42d9d","Type":"ContainerStarted","Data":"a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a"}
Jul 28 20:29:11.778933 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.778844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875" event={"ID":"73eb9621-6497-4009-bbdc-acfc29f42d9d","Type":"ContainerStarted","Data":"15f861f91d654acd13488b76e7460279c633ee345831cd76bb0465f46662e913"}
Jul 28 20:29:11.778973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.778945    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:11.793700 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.793647    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7" podStartSLOduration=1.7936317370000001 podStartE2EDuration="1.793631737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:11.792561087 +0000 UTC m=+1801.646872155" watchObservedRunningTime="2026-07-28 20:29:11.793631737 +0000 UTC m=+1801.647942802"
Jul 28 20:29:11.812203 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.812146    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875" podStartSLOduration=1.812131778 podStartE2EDuration="1.812131778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:11.810847764 +0000 UTC m=+1801.665158832" watchObservedRunningTime="2026-07-28 20:29:11.812131778 +0000 UTC m=+1801.666442887"
Jul 28 20:29:11.830594 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:11.830526    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf" podStartSLOduration=2.374745371 podStartE2EDuration="2.830510298s" totalImagesPullingTime="455.764927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:11.828378896 +0000 UTC m=+1801.682689989" watchObservedRunningTime="2026-07-28 20:29:11.830510298 +0000 UTC m=+1801.684821364"
Jul 28 20:29:17.788182 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:17.788150    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:22.784768 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:22.784726    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875" podUID="73eb9621-6497-4009-bbdc-acfc29f42d9d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.59:9004: connect: connection refused"
Jul 28 20:29:22.785277 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:22.785259    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:23.757273 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:23.757229    2582 patch_prober.go:28] interesting pod/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 20:29:23.757492 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:23.757326    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 20:29:23.839686 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:23.839600    2582 generic.go:356] "Generic (PLEG): container finished" podID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerID="8392be257192b34a7d8a7508e8025e2b3e11de3df95f1a57954382c7da551b95" exitCode=0
Jul 28 20:29:23.839686 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:23.839668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" event={"ID":"67f08c12-606b-4c6f-8f80-7b305d36799a","Type":"ContainerDied","Data":"8392be257192b34a7d8a7508e8025e2b3e11de3df95f1a57954382c7da551b95"}
Jul 28 20:29:23.840118 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:23.840104    2582 scope.go:117] "RemoveContainer" containerID="8392be257192b34a7d8a7508e8025e2b3e11de3df95f1a57954382c7da551b95"
Jul 28 20:29:23.841315 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:23.841269    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:29:23.841387 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:23.841371    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:29:23.841424 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:23.841414    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:29:24.852204 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:24.852166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" event={"ID":"67f08c12-606b-4c6f-8f80-7b305d36799a","Type":"ContainerStarted","Data":"05f1f4c43ae3cf773f89ed364aafae750eff10a29942a83c7caaae25673aeee6"}
Jul 28 20:29:25.589104 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:25.589068    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:25.589282 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:25.589232    2582 patch_prober.go:28] interesting pod/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 20:29:25.589362 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:25.589277    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 20:29:26.589892 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:26.589847    2582 patch_prober.go:28] interesting pod/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 20:29:26.590411 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:26.589897    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 20:29:27.167783 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:27.167720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:27.589962 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:27.589922    2582 patch_prober.go:28] interesting pod/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 20:29:27.590398 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:27.589981    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 20:29:28.589537 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:28.589501    2582 patch_prober.go:28] interesting pod/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 28 20:29:28.589752 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:28.589562    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 28 20:29:29.593895 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:29.593862    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:29.594345 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:29.594193    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:29.595022 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:29.595004    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:32.784186 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:32.784156    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:40.309347 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.309314    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"]
Jul 28 20:29:40.309960 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.309668    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf" podUID="c589ab26-04bd-408c-a036-5ea42f15fb1c" containerName="maas-api" containerID="cri-o://6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c" gracePeriod=30
Jul 28 20:29:40.341326 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.341272    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:40.413952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.413912    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"]
Jul 28 20:29:40.414178 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.414151    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7" podUID="352ce7d2-8323-4110-90d0-d6257494cb93" containerName="payload-processing" containerID="cri-o://da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3" gracePeriod=30
Jul 28 20:29:40.439211 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.439176    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"]
Jul 28 20:29:40.440182 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.439451    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875" podUID="73eb9621-6497-4009-bbdc-acfc29f42d9d" containerName="payload-pre-processing" containerID="cri-o://a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a" gracePeriod=30
Jul 28 20:29:40.504771 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.504732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:40.591577 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.591112    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:40.623205 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.623118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:40.715707 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.715682    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:40.721472 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.721447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c589ab26-04bd-408c-a036-5ea42f15fb1c-openshift-service-ca\") pod \"c589ab26-04bd-408c-a036-5ea42f15fb1c\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") "
Jul 28 20:29:40.721613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.721485    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:40.721613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.721509    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c589ab26-04bd-408c-a036-5ea42f15fb1c-maas-api-tls\") pod \"c589ab26-04bd-408c-a036-5ea42f15fb1c\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") "
Jul 28 20:29:40.721613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.721597    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9p6w\" (UniqueName: \"kubernetes.io/projected/c589ab26-04bd-408c-a036-5ea42f15fb1c-kube-api-access-p9p6w\") pod \"c589ab26-04bd-408c-a036-5ea42f15fb1c\" (UID: \"c589ab26-04bd-408c-a036-5ea42f15fb1c\") "
Jul 28 20:29:40.722257 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.721850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c589ab26-04bd-408c-a036-5ea42f15fb1c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c589ab26-04bd-408c-a036-5ea42f15fb1c" (UID: "c589ab26-04bd-408c-a036-5ea42f15fb1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:29:40.729926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.724402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:40.729926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.725885    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c589ab26-04bd-408c-a036-5ea42f15fb1c-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:40.729926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.729787    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c589ab26-04bd-408c-a036-5ea42f15fb1c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c589ab26-04bd-408c-a036-5ea42f15fb1c" (UID: "c589ab26-04bd-408c-a036-5ea42f15fb1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:29:40.730175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.729999    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c589ab26-04bd-408c-a036-5ea42f15fb1c-kube-api-access-p9p6w" (OuterVolumeSpecName: "kube-api-access-p9p6w") pod "c589ab26-04bd-408c-a036-5ea42f15fb1c" (UID: "c589ab26-04bd-408c-a036-5ea42f15fb1c"). InnerVolumeSpecName "kube-api-access-p9p6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:29:40.827153 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.827061    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352ce7d2-8323-4110-90d0-d6257494cb93-plugins-config-volume\") pod \"352ce7d2-8323-4110-90d0-d6257494cb93\" (UID: \"352ce7d2-8323-4110-90d0-d6257494cb93\") "
Jul 28 20:29:40.827665 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.827637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73eb9621-6497-4009-bbdc-acfc29f42d9d-plugins-config-volume\") pod \"73eb9621-6497-4009-bbdc-acfc29f42d9d\" (UID: \"73eb9621-6497-4009-bbdc-acfc29f42d9d\") "
Jul 28 20:29:40.827807 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.827697    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb5vf\" (UniqueName: \"kubernetes.io/projected/352ce7d2-8323-4110-90d0-d6257494cb93-kube-api-access-pb5vf\") pod \"352ce7d2-8323-4110-90d0-d6257494cb93\" (UID: \"352ce7d2-8323-4110-90d0-d6257494cb93\") "
Jul 28 20:29:40.827807 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.827735    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm6x4\" (UniqueName: \"kubernetes.io/projected/73eb9621-6497-4009-bbdc-acfc29f42d9d-kube-api-access-vm6x4\") pod \"73eb9621-6497-4009-bbdc-acfc29f42d9d\" (UID: \"73eb9621-6497-4009-bbdc-acfc29f42d9d\") "
Jul 28 20:29:40.828020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.828005    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9p6w\" (UniqueName: \"kubernetes.io/projected/c589ab26-04bd-408c-a036-5ea42f15fb1c-kube-api-access-p9p6w\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:40.828081 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.828027    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c589ab26-04bd-408c-a036-5ea42f15fb1c-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:40.828733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.827516    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352ce7d2-8323-4110-90d0-d6257494cb93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "352ce7d2-8323-4110-90d0-d6257494cb93" (UID: "352ce7d2-8323-4110-90d0-d6257494cb93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:29:40.828888 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.828581    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73eb9621-6497-4009-bbdc-acfc29f42d9d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73eb9621-6497-4009-bbdc-acfc29f42d9d" (UID: "73eb9621-6497-4009-bbdc-acfc29f42d9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:29:40.831337 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.830576    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73eb9621-6497-4009-bbdc-acfc29f42d9d-kube-api-access-vm6x4" (OuterVolumeSpecName: "kube-api-access-vm6x4") pod "73eb9621-6497-4009-bbdc-acfc29f42d9d" (UID: "73eb9621-6497-4009-bbdc-acfc29f42d9d"). InnerVolumeSpecName "kube-api-access-vm6x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:29:40.831337 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.830870    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352ce7d2-8323-4110-90d0-d6257494cb93-kube-api-access-pb5vf" (OuterVolumeSpecName: "kube-api-access-pb5vf") pod "352ce7d2-8323-4110-90d0-d6257494cb93" (UID: "352ce7d2-8323-4110-90d0-d6257494cb93"). InnerVolumeSpecName "kube-api-access-pb5vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:29:40.838458 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.838431    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"]
Jul 28 20:29:40.838699 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.838657    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy" containerID="cri-o://05f1f4c43ae3cf773f89ed364aafae750eff10a29942a83c7caaae25673aeee6" gracePeriod=30
Jul 28 20:29:40.869356 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.869317    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:40.929265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.929226    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6x4\" (UniqueName: \"kubernetes.io/projected/73eb9621-6497-4009-bbdc-acfc29f42d9d-kube-api-access-vm6x4\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:40.929265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.929265    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352ce7d2-8323-4110-90d0-d6257494cb93-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:40.929490 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.929281    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73eb9621-6497-4009-bbdc-acfc29f42d9d-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:40.929490 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.929310    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb5vf\" (UniqueName: \"kubernetes.io/projected/352ce7d2-8323-4110-90d0-d6257494cb93-kube-api-access-pb5vf\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:40.935029 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.934997    2582 generic.go:356] "Generic (PLEG): container finished" podID="352ce7d2-8323-4110-90d0-d6257494cb93" containerID="da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3" exitCode=0
Jul 28 20:29:40.935220 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.935057    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"
Jul 28 20:29:40.935220 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.935071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7" event={"ID":"352ce7d2-8323-4110-90d0-d6257494cb93","Type":"ContainerDied","Data":"da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3"}
Jul 28 20:29:40.935220 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.935121    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7" event={"ID":"352ce7d2-8323-4110-90d0-d6257494cb93","Type":"ContainerDied","Data":"1b8b5246926e552ec2461debd6b0151e27d683df2be3105f25c7780e9a9ef1fa"}
Jul 28 20:29:40.935220 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.935144    2582 scope.go:117] "RemoveContainer" containerID="da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3"
Jul 28 20:29:40.936476 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.936449    2582 generic.go:356] "Generic (PLEG): container finished" podID="c589ab26-04bd-408c-a036-5ea42f15fb1c" containerID="6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c" exitCode=0
Jul 28 20:29:40.936592 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.936516    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"
Jul 28 20:29:40.936592 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.936520    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf" event={"ID":"c589ab26-04bd-408c-a036-5ea42f15fb1c","Type":"ContainerDied","Data":"6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c"}
Jul 28 20:29:40.936592 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.936566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf" event={"ID":"c589ab26-04bd-408c-a036-5ea42f15fb1c","Type":"ContainerDied","Data":"533d3bcbee332f58647dbaab33c3bb38d379018109040e331ac1170e0ab06aa3"}
Jul 28 20:29:40.941572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.938165    2582 generic.go:356] "Generic (PLEG): container finished" podID="73eb9621-6497-4009-bbdc-acfc29f42d9d" containerID="a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a" exitCode=0
Jul 28 20:29:40.941572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.938207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875" event={"ID":"73eb9621-6497-4009-bbdc-acfc29f42d9d","Type":"ContainerDied","Data":"a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a"}
Jul 28 20:29:40.941572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.938244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875" event={"ID":"73eb9621-6497-4009-bbdc-acfc29f42d9d","Type":"ContainerDied","Data":"15f861f91d654acd13488b76e7460279c633ee345831cd76bb0465f46662e913"}
Jul 28 20:29:40.941572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.938351    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"
Jul 28 20:29:40.947193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.947049    2582 scope.go:117] "RemoveContainer" containerID="da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3"
Jul 28 20:29:40.947480 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:40.947450    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3\": container with ID starting with da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3 not found: ID does not exist" containerID="da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3"
Jul 28 20:29:40.947606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.947491    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3"} err="failed to get container status \"da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3\": rpc error: code = NotFound desc = could not find container \"da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3\": container with ID starting with da0221f9608d86cefcbeedc83de2ea9b9d21f3eded1acd855f1bad8ff22385e3 not found: ID does not exist"
Jul 28 20:29:40.947606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.947515    2582 scope.go:117] "RemoveContainer" containerID="6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c"
Jul 28 20:29:40.957927 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.957906    2582 scope.go:117] "RemoveContainer" containerID="6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c"
Jul 28 20:29:40.958223 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:40.958200    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c\": container with ID starting with 6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c not found: ID does not exist" containerID="6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c"
Jul 28 20:29:40.958359 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.958231    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c"} err="failed to get container status \"6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c\": rpc error: code = NotFound desc = could not find container \"6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c\": container with ID starting with 6a5be96cd025e86598bd6b97a5c7a823e138be166d3a6cd6fa3f8c2627fc992c not found: ID does not exist"
Jul 28 20:29:40.958359 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.958248    2582 scope.go:117] "RemoveContainer" containerID="a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a"
Jul 28 20:29:40.968901 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.968883    2582 scope.go:117] "RemoveContainer" containerID="a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a"
Jul 28 20:29:40.969100 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:40.969080    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a\": container with ID starting with a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a not found: ID does not exist" containerID="a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a"
Jul 28 20:29:40.969200 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.969105    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a"} err="failed to get container status \"a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a\": rpc error: code = NotFound desc = could not find container \"a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a\": container with ID starting with a46ba28f94c67b78387f7f1b521a1ef8fbbaf60c6211e88db2333eb14ca26b4a not found: ID does not exist"
Jul 28 20:29:40.979792 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.979763    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"]
Jul 28 20:29:40.982473 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.982448    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-addc0a70-7cff46b748-znrxf"]
Jul 28 20:29:40.995918 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:40.995894    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"]
Jul 28 20:29:41.002174 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:41.002151    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-addc0a70-75d7d8fd9-s9mm7"]
Jul 28 20:29:41.015002 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:41.014959    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"]
Jul 28 20:29:41.021439 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:41.021412    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-addc0a70-6dbbf5664-v7875"]
Jul 28 20:29:41.970359 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:41.970321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:42.585800 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:42.585763    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="352ce7d2-8323-4110-90d0-d6257494cb93" path="/var/lib/kubelet/pods/352ce7d2-8323-4110-90d0-d6257494cb93/volumes"
Jul 28 20:29:42.586142 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:42.586128    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73eb9621-6497-4009-bbdc-acfc29f42d9d" path="/var/lib/kubelet/pods/73eb9621-6497-4009-bbdc-acfc29f42d9d/volumes"
Jul 28 20:29:42.586498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:42.586486    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c589ab26-04bd-408c-a036-5ea42f15fb1c" path="/var/lib/kubelet/pods/c589ab26-04bd-408c-a036-5ea42f15fb1c/volumes"
Jul 28 20:29:42.738443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:42.738411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:44.123396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.123364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:44.220260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220227    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"]
Jul 28 20:29:44.220630 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220617    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="352ce7d2-8323-4110-90d0-d6257494cb93" containerName="payload-processing"
Jul 28 20:29:44.220678 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220631    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="352ce7d2-8323-4110-90d0-d6257494cb93" containerName="payload-processing"
Jul 28 20:29:44.220678 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220644    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c589ab26-04bd-408c-a036-5ea42f15fb1c" containerName="maas-api"
Jul 28 20:29:44.220678 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220649    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c589ab26-04bd-408c-a036-5ea42f15fb1c" containerName="maas-api"
Jul 28 20:29:44.220678 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220659    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73eb9621-6497-4009-bbdc-acfc29f42d9d" containerName="payload-pre-processing"
Jul 28 20:29:44.220678 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220665    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="73eb9621-6497-4009-bbdc-acfc29f42d9d" containerName="payload-pre-processing"
Jul 28 20:29:44.220831 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220719    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="352ce7d2-8323-4110-90d0-d6257494cb93" containerName="payload-processing"
Jul 28 20:29:44.220831 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220727    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c589ab26-04bd-408c-a036-5ea42f15fb1c" containerName="maas-api"
Jul 28 20:29:44.220831 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.220733    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="73eb9621-6497-4009-bbdc-acfc29f42d9d" containerName="payload-pre-processing"
Jul 28 20:29:44.223867 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.223611    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.229835 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.229810    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-89b18cb8-serving-cert\""
Jul 28 20:29:44.245828 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.245801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"]
Jul 28 20:29:44.251782 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.251730    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:44.340616 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.340581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:44.360716 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.360680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqkqh\" (UniqueName: \"kubernetes.io/projected/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-kube-api-access-wqkqh\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.360893 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.360834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-openshift-service-ca\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.360967 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.360898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-maas-api-tls\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.462494 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.462400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-openshift-service-ca\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.462494 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.462452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-maas-api-tls\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.462696 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.462499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqkqh\" (UniqueName: \"kubernetes.io/projected/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-kube-api-access-wqkqh\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.462696 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:44.462628    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-serving-cert: secret "maas-api-e2e-ait-89b18cb8-serving-cert" not found
Jul 28 20:29:44.462785 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:29:44.462705    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-maas-api-tls podName:ff275438-f1d5-48e5-bdcc-0a76eca2d5b0 nodeName:}" failed. No retries permitted until 2026-07-28 20:29:44.962688436 +0000 UTC m=+1834.816999480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-maas-api-tls") pod "maas-api-e2e-ait-89b18cb8-667f45885b-246l8" (UID: "ff275438-f1d5-48e5-bdcc-0a76eca2d5b0") : secret "maas-api-e2e-ait-89b18cb8-serving-cert" not found
Jul 28 20:29:44.463129 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.463107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-openshift-service-ca\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.479777 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.479745    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqkqh\" (UniqueName: \"kubernetes.io/projected/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-kube-api-access-wqkqh\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.596509 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.596475    2582 patch_prober.go:28] interesting pod/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:29:44.596746 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.596543    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:29:44.669975 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.669935    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:44.875618 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.875583    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"]
Jul 28 20:29:44.879193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.879176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:29:44.882116 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.882095    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-89b18cb8\""
Jul 28 20:29:44.888271 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.888244    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"]
Jul 28 20:29:44.917608 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.917579    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"]
Jul 28 20:29:44.920898 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.920882    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:44.923899 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.923877    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-89b18cb8-dockercfg-nh9nl\""
Jul 28 20:29:44.945495 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.945469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"]
Jul 28 20:29:44.967540 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.967515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-maas-api-tls\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:44.970353 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:44.970326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-maas-api-tls\") pod \"maas-api-e2e-ait-89b18cb8-667f45885b-246l8\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:45.068255 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.068212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad88daa8-70ac-4f3f-92a7-218e484228a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7\" (UID: \"ad88daa8-70ac-4f3f-92a7-218e484228a9\") " pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:45.068255 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.068259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj\" (UID: \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:29:45.068496 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.068285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzpcb\" (UniqueName: \"kubernetes.io/projected/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-kube-api-access-zzpcb\") pod \"payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj\" (UID: \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:29:45.068496 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.068392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9n8h\" (UniqueName: \"kubernetes.io/projected/ad88daa8-70ac-4f3f-92a7-218e484228a9-kube-api-access-j9n8h\") pod \"payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7\" (UID: \"ad88daa8-70ac-4f3f-92a7-218e484228a9\") " pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:45.137403 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.137265    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:45.169471 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.169437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpcb\" (UniqueName: \"kubernetes.io/projected/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-kube-api-access-zzpcb\") pod \"payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj\" (UID: \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:29:45.169666 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.169492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9n8h\" (UniqueName: \"kubernetes.io/projected/ad88daa8-70ac-4f3f-92a7-218e484228a9-kube-api-access-j9n8h\") pod \"payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7\" (UID: \"ad88daa8-70ac-4f3f-92a7-218e484228a9\") " pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:45.169666 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.169561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad88daa8-70ac-4f3f-92a7-218e484228a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7\" (UID: \"ad88daa8-70ac-4f3f-92a7-218e484228a9\") " pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:45.169666 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.169591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj\" (UID: \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:29:45.170491 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.170467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad88daa8-70ac-4f3f-92a7-218e484228a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7\" (UID: \"ad88daa8-70ac-4f3f-92a7-218e484228a9\") " pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:45.170565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.170467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj\" (UID: \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:29:45.183387 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.183352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9n8h\" (UniqueName: \"kubernetes.io/projected/ad88daa8-70ac-4f3f-92a7-218e484228a9-kube-api-access-j9n8h\") pod \"payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7\" (UID: \"ad88daa8-70ac-4f3f-92a7-218e484228a9\") " pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:45.183554 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.183421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzpcb\" (UniqueName: \"kubernetes.io/projected/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-kube-api-access-zzpcb\") pod \"payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj\" (UID: \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:29:45.191308 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.191179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:29:45.234681 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.234378    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:45.318587 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.318542    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"]
Jul 28 20:29:45.375286 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.375249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"]
Jul 28 20:29:45.378728 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:29:45.378693    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526f6c32_3cb8_41cd_bcb0_59dd47ae97f7.slice/crio-30e895bdafada5cebcab8f7df6fa0c8a95b5d8ece5b4eece8fdb16e9853e2585 WatchSource:0}: Error finding container 30e895bdafada5cebcab8f7df6fa0c8a95b5d8ece5b4eece8fdb16e9853e2585: Status 404 returned error can't find the container with id 30e895bdafada5cebcab8f7df6fa0c8a95b5d8ece5b4eece8fdb16e9853e2585
Jul 28 20:29:45.499931 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.499886    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:45.612187 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.612145    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"]
Jul 28 20:29:45.612435 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.612407    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:45.614201 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:29:45.614174    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad88daa8_70ac_4f3f_92a7_218e484228a9.slice/crio-a3620b93076ca14770ddfb6138e70ba69b623664b7d1b08e4e2c5de416b29dab WatchSource:0}: Error finding container a3620b93076ca14770ddfb6138e70ba69b623664b7d1b08e4e2c5de416b29dab: Status 404 returned error can't find the container with id a3620b93076ca14770ddfb6138e70ba69b623664b7d1b08e4e2c5de416b29dab
Jul 28 20:29:45.971613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.971574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj" event={"ID":"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7","Type":"ContainerStarted","Data":"b41335be7bf5ca477957c63c22b9c4b03d407bc61d98fa52a8e41c289390506f"}
Jul 28 20:29:45.971797 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.971636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj" event={"ID":"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7","Type":"ContainerStarted","Data":"30e895bdafada5cebcab8f7df6fa0c8a95b5d8ece5b4eece8fdb16e9853e2585"}
Jul 28 20:29:45.971797 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.971671    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:29:45.973990 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.973965    2582 generic.go:356] "Generic (PLEG): container finished" podID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerID="05f1f4c43ae3cf773f89ed364aafae750eff10a29942a83c7caaae25673aeee6" exitCode=0
Jul 28 20:29:45.974114 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.974048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" event={"ID":"67f08c12-606b-4c6f-8f80-7b305d36799a","Type":"ContainerDied","Data":"05f1f4c43ae3cf773f89ed364aafae750eff10a29942a83c7caaae25673aeee6"}
Jul 28 20:29:45.974175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.974117    2582 scope.go:117] "RemoveContainer" containerID="8392be257192b34a7d8a7508e8025e2b3e11de3df95f1a57954382c7da551b95"
Jul 28 20:29:45.975731 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.975708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7" event={"ID":"ad88daa8-70ac-4f3f-92a7-218e484228a9","Type":"ContainerStarted","Data":"5cd3db5922a9abdf6cd48682348b039d8625f633b9b50b94fe10e2bb60afd793"}
Jul 28 20:29:45.975847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.975763    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:45.975847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.975786    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7" event={"ID":"ad88daa8-70ac-4f3f-92a7-218e484228a9","Type":"ContainerStarted","Data":"a3620b93076ca14770ddfb6138e70ba69b623664b7d1b08e4e2c5de416b29dab"}
Jul 28 20:29:45.977013 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:45.976988    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8" event={"ID":"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0","Type":"ContainerStarted","Data":"e6ad01c93fc9f066908a589337bf98c1cc466a2a348a84b0797feeaea2a8fe92"}
Jul 28 20:29:46.003745 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.003697    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj" podStartSLOduration=2.003681274 podStartE2EDuration="2.003681274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:46.003113477 +0000 UTC m=+1835.857424540" watchObservedRunningTime="2026-07-28 20:29:46.003681274 +0000 UTC m=+1835.857992339"
Jul 28 20:29:46.052802 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.052734    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7" podStartSLOduration=2.052710464 podStartE2EDuration="2.052710464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:46.052155638 +0000 UTC m=+1835.906466717" watchObservedRunningTime="2026-07-28 20:29:46.052710464 +0000 UTC m=+1835.907021532"
Jul 28 20:29:46.101369 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.101342    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:46.280747 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.280715    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-credential-socket\") pod \"67f08c12-606b-4c6f-8f80-7b305d36799a\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") "
Jul 28 20:29:46.281222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.280759    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7vnv\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-kube-api-access-c7vnv\") pod \"67f08c12-606b-4c6f-8f80-7b305d36799a\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") "
Jul 28 20:29:46.281222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.280787    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-data\") pod \"67f08c12-606b-4c6f-8f80-7b305d36799a\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") "
Jul 28 20:29:46.281222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.280813    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-socket\") pod \"67f08c12-606b-4c6f-8f80-7b305d36799a\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") "
Jul 28 20:29:46.281222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.280833    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-token\") pod \"67f08c12-606b-4c6f-8f80-7b305d36799a\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") "
Jul 28 20:29:46.281222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.280854    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f08c12-606b-4c6f-8f80-7b305d36799a-istiod-ca-cert\") pod \"67f08c12-606b-4c6f-8f80-7b305d36799a\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") "
Jul 28 20:29:46.281222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.280871    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-envoy\") pod \"67f08c12-606b-4c6f-8f80-7b305d36799a\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") "
Jul 28 20:29:46.281222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.280972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-certs\") pod \"67f08c12-606b-4c6f-8f80-7b305d36799a\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") "
Jul 28 20:29:46.281222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.281008    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-podinfo\") pod \"67f08c12-606b-4c6f-8f80-7b305d36799a\" (UID: \"67f08c12-606b-4c6f-8f80-7b305d36799a\") "
Jul 28 20:29:46.281222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.281047    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67f08c12-606b-4c6f-8f80-7b305d36799a" (UID: "67f08c12-606b-4c6f-8f80-7b305d36799a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:29:46.281730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.281226    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67f08c12-606b-4c6f-8f80-7b305d36799a" (UID: "67f08c12-606b-4c6f-8f80-7b305d36799a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:29:46.281730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.281237    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67f08c12-606b-4c6f-8f80-7b305d36799a" (UID: "67f08c12-606b-4c6f-8f80-7b305d36799a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:29:46.281730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.281339    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-credential-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:46.281730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.281357    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:46.281730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.281372    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-workload-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:46.281730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.281385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f08c12-606b-4c6f-8f80-7b305d36799a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67f08c12-606b-4c6f-8f80-7b305d36799a" (UID: "67f08c12-606b-4c6f-8f80-7b305d36799a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:29:46.281730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.281673    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-data" (OuterVolumeSpecName: "istio-data") pod "67f08c12-606b-4c6f-8f80-7b305d36799a" (UID: "67f08c12-606b-4c6f-8f80-7b305d36799a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:29:46.283152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.283123    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-kube-api-access-c7vnv" (OuterVolumeSpecName: "kube-api-access-c7vnv") pod "67f08c12-606b-4c6f-8f80-7b305d36799a" (UID: "67f08c12-606b-4c6f-8f80-7b305d36799a"). InnerVolumeSpecName "kube-api-access-c7vnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:29:46.283265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.283157    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-token" (OuterVolumeSpecName: "istio-token") pod "67f08c12-606b-4c6f-8f80-7b305d36799a" (UID: "67f08c12-606b-4c6f-8f80-7b305d36799a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:29:46.283265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.283159    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67f08c12-606b-4c6f-8f80-7b305d36799a" (UID: "67f08c12-606b-4c6f-8f80-7b305d36799a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:29:46.283265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.283158    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67f08c12-606b-4c6f-8f80-7b305d36799a" (UID: "67f08c12-606b-4c6f-8f80-7b305d36799a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:29:46.382284 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.382246    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:46.382284 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.382277    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f08c12-606b-4c6f-8f80-7b305d36799a-istiod-ca-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:46.382498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.382314    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-envoy\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:46.382498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.382326    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-podinfo\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:46.382498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.382338    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7vnv\" (UniqueName: \"kubernetes.io/projected/67f08c12-606b-4c6f-8f80-7b305d36799a-kube-api-access-c7vnv\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:46.382498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.382351    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f08c12-606b-4c6f-8f80-7b305d36799a-istio-data\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:29:46.983197 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.983167    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"
Jul 28 20:29:46.983415 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.983198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p" event={"ID":"67f08c12-606b-4c6f-8f80-7b305d36799a","Type":"ContainerDied","Data":"5c631c39b042624c5199cdf67c80134c502fe07eea920245ffcf071956a3c1f0"}
Jul 28 20:29:46.983415 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.983308    2582 scope.go:117] "RemoveContainer" containerID="05f1f4c43ae3cf773f89ed364aafae750eff10a29942a83c7caaae25673aeee6"
Jul 28 20:29:46.984943 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:46.984916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8" event={"ID":"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0","Type":"ContainerStarted","Data":"b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853"}
Jul 28 20:29:47.005044 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:47.005017    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"]
Jul 28 20:29:47.017287 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:47.017261    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-addc0a70-gw-openshift-default-6cc8bc9c7b-44p2p"]
Jul 28 20:29:47.050108 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:47.050059    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8" podStartSLOduration=1.978049157 podStartE2EDuration="3.050042723s" totalImagesPullingTime="1.071993566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:47.049860907 +0000 UTC m=+1836.904172006" watchObservedRunningTime="2026-07-28 20:29:47.050042723 +0000 UTC m=+1836.904353788"
Jul 28 20:29:47.990335 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:47.990305    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:48.583479 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:48.583448    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" path="/var/lib/kubelet/pods/67f08c12-606b-4c6f-8f80-7b305d36799a/volumes"
Jul 28 20:29:54.002151 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:54.002119    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:29:56.987420 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:56.987388    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:29:56.987804 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:29:56.987458    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:30:09.754413 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.754378    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"]
Jul 28 20:30:09.754793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.754766    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy"
Jul 28 20:30:09.754793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.754777    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy"
Jul 28 20:30:09.754793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.754785    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy"
Jul 28 20:30:09.754793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.754791    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy"
Jul 28 20:30:09.754916 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.754847    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy"
Jul 28 20:30:09.759639 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.759620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"
Jul 28 20:30:09.762494 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.762466    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\""
Jul 28 20:30:09.765545 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.765520    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"]
Jul 28 20:30:09.886938 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.886896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283528f9-0156-43d2-9675-427f128f1f00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k\" (UID: \"283528f9-0156-43d2-9675-427f128f1f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"
Jul 28 20:30:09.988169 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.988133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283528f9-0156-43d2-9675-427f128f1f00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k\" (UID: \"283528f9-0156-43d2-9675-427f128f1f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"
Jul 28 20:30:09.988791 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:09.988771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283528f9-0156-43d2-9675-427f128f1f00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k\" (UID: \"283528f9-0156-43d2-9675-427f128f1f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"
Jul 28 20:30:10.072426 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:10.072338    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"
Jul 28 20:30:10.208789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:10.208760    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"]
Jul 28 20:30:10.211168 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:30:10.211138    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283528f9_0156_43d2_9675_427f128f1f00.slice/crio-61450b345b65e5ce36970df8b7522b12d09dc62ef4ac69578d33d72676e34c19 WatchSource:0}: Error finding container 61450b345b65e5ce36970df8b7522b12d09dc62ef4ac69578d33d72676e34c19: Status 404 returned error can't find the container with id 61450b345b65e5ce36970df8b7522b12d09dc62ef4ac69578d33d72676e34c19
Jul 28 20:30:11.112212 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:11.112174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k" event={"ID":"283528f9-0156-43d2-9675-427f128f1f00","Type":"ContainerStarted","Data":"61450b345b65e5ce36970df8b7522b12d09dc62ef4ac69578d33d72676e34c19"}
Jul 28 20:30:14.128834 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:14.128796    2582 generic.go:356] "Generic (PLEG): container finished" podID="283528f9-0156-43d2-9675-427f128f1f00" containerID="9fc0fabc01d37e12649e6418ba2cf728103c3daad50c656852411c8d9fe86522" exitCode=0
Jul 28 20:30:14.129257 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:14.128886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k" event={"ID":"283528f9-0156-43d2-9675-427f128f1f00","Type":"ContainerDied","Data":"9fc0fabc01d37e12649e6418ba2cf728103c3daad50c656852411c8d9fe86522"}
Jul 28 20:30:15.281592 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:15.281564    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"
Jul 28 20:30:15.341476 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:15.341437    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283528f9-0156-43d2-9675-427f128f1f00-maas-api-service-ca\") pod \"283528f9-0156-43d2-9675-427f128f1f00\" (UID: \"283528f9-0156-43d2-9675-427f128f1f00\") "
Jul 28 20:30:15.341808 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:15.341786    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283528f9-0156-43d2-9675-427f128f1f00-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "283528f9-0156-43d2-9675-427f128f1f00" (UID: "283528f9-0156-43d2-9675-427f128f1f00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:30:15.442503 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:15.442420    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283528f9-0156-43d2-9675-427f128f1f00-maas-api-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:30:16.148879 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:16.148850    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"
Jul 28 20:30:16.149076 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:16.148840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k" event={"ID":"283528f9-0156-43d2-9675-427f128f1f00","Type":"ContainerDied","Data":"61450b345b65e5ce36970df8b7522b12d09dc62ef4ac69578d33d72676e34c19"}
Jul 28 20:30:16.149076 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:16.148991    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61450b345b65e5ce36970df8b7522b12d09dc62ef4ac69578d33d72676e34c19"
Jul 28 20:30:24.822279 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:24.822240    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"]
Jul 28 20:30:24.823119 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:24.823083    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8" podUID="ff275438-f1d5-48e5-bdcc-0a76eca2d5b0" containerName="maas-api" containerID="cri-o://b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853" gracePeriod=30
Jul 28 20:30:24.863948 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:24.863916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:24.985720 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:24.985685    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"]
Jul 28 20:30:24.985952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:24.985925    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7" podUID="ad88daa8-70ac-4f3f-92a7-218e484228a9" containerName="payload-processing" containerID="cri-o://5cd3db5922a9abdf6cd48682348b039d8625f633b9b50b94fe10e2bb60afd793" gracePeriod=30
Jul 28 20:30:25.003813 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.003781    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"]
Jul 28 20:30:25.004192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.004165    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj" podUID="526f6c32-3cb8-41cd-bcb0-59dd47ae97f7" containerName="payload-pre-processing" containerID="cri-o://b41335be7bf5ca477957c63c22b9c4b03d407bc61d98fa52a8e41c289390506f" gracePeriod=30
Jul 28 20:30:25.048259 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.048216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:25.129193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.129168    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:30:25.187952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.187866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:25.194645 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.194619    2582 generic.go:356] "Generic (PLEG): container finished" podID="ad88daa8-70ac-4f3f-92a7-218e484228a9" containerID="5cd3db5922a9abdf6cd48682348b039d8625f633b9b50b94fe10e2bb60afd793" exitCode=0
Jul 28 20:30:25.194736 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.194688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7" event={"ID":"ad88daa8-70ac-4f3f-92a7-218e484228a9","Type":"ContainerDied","Data":"5cd3db5922a9abdf6cd48682348b039d8625f633b9b50b94fe10e2bb60afd793"}
Jul 28 20:30:25.196105 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.196081    2582 generic.go:356] "Generic (PLEG): container finished" podID="ff275438-f1d5-48e5-bdcc-0a76eca2d5b0" containerID="b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853" exitCode=0
Jul 28 20:30:25.196161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.196117    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8" event={"ID":"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0","Type":"ContainerDied","Data":"b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853"}
Jul 28 20:30:25.196161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.196134    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"
Jul 28 20:30:25.196235 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.196171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8" event={"ID":"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0","Type":"ContainerDied","Data":"e6ad01c93fc9f066908a589337bf98c1cc466a2a348a84b0797feeaea2a8fe92"}
Jul 28 20:30:25.196235 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.196202    2582 scope.go:117] "RemoveContainer" containerID="b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853"
Jul 28 20:30:25.198234 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.198196    2582 generic.go:356] "Generic (PLEG): container finished" podID="526f6c32-3cb8-41cd-bcb0-59dd47ae97f7" containerID="b41335be7bf5ca477957c63c22b9c4b03d407bc61d98fa52a8e41c289390506f" exitCode=0
Jul 28 20:30:25.198369 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.198228    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj" event={"ID":"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7","Type":"ContainerDied","Data":"b41335be7bf5ca477957c63c22b9c4b03d407bc61d98fa52a8e41c289390506f"}
Jul 28 20:30:25.206268 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.206244    2582 scope.go:117] "RemoveContainer" containerID="b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853"
Jul 28 20:30:25.206623 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:30:25.206603    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853\": container with ID starting with b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853 not found: ID does not exist" containerID="b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853"
Jul 28 20:30:25.206697 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.206638    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853"} err="failed to get container status \"b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853\": rpc error: code = NotFound desc = could not find container \"b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853\": container with ID starting with b8cd4d98fddeed905bb24938d7d904dc4e5231d8e637d2e04362cdf2569c5853 not found: ID does not exist"
Jul 28 20:30:25.229914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.229152    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqkqh\" (UniqueName: \"kubernetes.io/projected/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-kube-api-access-wqkqh\") pod \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") "
Jul 28 20:30:25.229914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.229209    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-openshift-service-ca\") pod \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") "
Jul 28 20:30:25.229914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.229261    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-maas-api-tls\") pod \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\" (UID: \"ff275438-f1d5-48e5-bdcc-0a76eca2d5b0\") "
Jul 28 20:30:25.230336 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.230080    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff275438-f1d5-48e5-bdcc-0a76eca2d5b0" (UID: "ff275438-f1d5-48e5-bdcc-0a76eca2d5b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:30:25.232087 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.232039    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-kube-api-access-wqkqh" (OuterVolumeSpecName: "kube-api-access-wqkqh") pod "ff275438-f1d5-48e5-bdcc-0a76eca2d5b0" (UID: "ff275438-f1d5-48e5-bdcc-0a76eca2d5b0"). InnerVolumeSpecName "kube-api-access-wqkqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:30:25.234330 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.234266    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff275438-f1d5-48e5-bdcc-0a76eca2d5b0" (UID: "ff275438-f1d5-48e5-bdcc-0a76eca2d5b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:30:25.285569 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.285532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:25.291521 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.291499    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:30:25.296102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.296076    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"]
Jul 28 20:30:25.297657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.297638    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:30:25.301570 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.301541    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89b18cb8-ws94k"]
Jul 28 20:30:25.330599 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.330569    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqkqh\" (UniqueName: \"kubernetes.io/projected/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-kube-api-access-wqkqh\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:30:25.330599 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.330596    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:30:25.330599 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.330609    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:30:25.431285 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.431193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad88daa8-70ac-4f3f-92a7-218e484228a9-plugins-config-volume\") pod \"ad88daa8-70ac-4f3f-92a7-218e484228a9\" (UID: \"ad88daa8-70ac-4f3f-92a7-218e484228a9\") "
Jul 28 20:30:25.431285 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.431260    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9n8h\" (UniqueName: \"kubernetes.io/projected/ad88daa8-70ac-4f3f-92a7-218e484228a9-kube-api-access-j9n8h\") pod \"ad88daa8-70ac-4f3f-92a7-218e484228a9\" (UID: \"ad88daa8-70ac-4f3f-92a7-218e484228a9\") "
Jul 28 20:30:25.431525 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.431318    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzpcb\" (UniqueName: \"kubernetes.io/projected/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-kube-api-access-zzpcb\") pod \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\" (UID: \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\") "
Jul 28 20:30:25.431525 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.431400    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-plugins-config-volume\") pod \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\" (UID: \"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7\") "
Jul 28 20:30:25.431679 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.431653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad88daa8-70ac-4f3f-92a7-218e484228a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad88daa8-70ac-4f3f-92a7-218e484228a9" (UID: "ad88daa8-70ac-4f3f-92a7-218e484228a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:30:25.431753 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.431737    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad88daa8-70ac-4f3f-92a7-218e484228a9-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:30:25.431894 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.431868    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "526f6c32-3cb8-41cd-bcb0-59dd47ae97f7" (UID: "526f6c32-3cb8-41cd-bcb0-59dd47ae97f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:30:25.433490 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.433463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad88daa8-70ac-4f3f-92a7-218e484228a9-kube-api-access-j9n8h" (OuterVolumeSpecName: "kube-api-access-j9n8h") pod "ad88daa8-70ac-4f3f-92a7-218e484228a9" (UID: "ad88daa8-70ac-4f3f-92a7-218e484228a9"). InnerVolumeSpecName "kube-api-access-j9n8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:30:25.433593 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.433577    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-kube-api-access-zzpcb" (OuterVolumeSpecName: "kube-api-access-zzpcb") pod "526f6c32-3cb8-41cd-bcb0-59dd47ae97f7" (UID: "526f6c32-3cb8-41cd-bcb0-59dd47ae97f7"). InnerVolumeSpecName "kube-api-access-zzpcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:30:25.522829 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.522793    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"]
Jul 28 20:30:25.532372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.532344    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9n8h\" (UniqueName: \"kubernetes.io/projected/ad88daa8-70ac-4f3f-92a7-218e484228a9-kube-api-access-j9n8h\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:30:25.532372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.532372    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzpcb\" (UniqueName: \"kubernetes.io/projected/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-kube-api-access-zzpcb\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:30:25.532559 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.532389    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:30:25.535008 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:25.534978    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89b18cb8-667f45885b-246l8"]
Jul 28 20:30:26.205083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.204984    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"
Jul 28 20:30:26.205083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.204982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7" event={"ID":"ad88daa8-70ac-4f3f-92a7-218e484228a9","Type":"ContainerDied","Data":"a3620b93076ca14770ddfb6138e70ba69b623664b7d1b08e4e2c5de416b29dab"}
Jul 28 20:30:26.205617 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.205109    2582 scope.go:117] "RemoveContainer" containerID="5cd3db5922a9abdf6cd48682348b039d8625f633b9b50b94fe10e2bb60afd793"
Jul 28 20:30:26.207549 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.207523    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"
Jul 28 20:30:26.207675 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.207534    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj" event={"ID":"526f6c32-3cb8-41cd-bcb0-59dd47ae97f7","Type":"ContainerDied","Data":"30e895bdafada5cebcab8f7df6fa0c8a95b5d8ece5b4eece8fdb16e9853e2585"}
Jul 28 20:30:26.214088 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.214069    2582 scope.go:117] "RemoveContainer" containerID="b41335be7bf5ca477957c63c22b9c4b03d407bc61d98fa52a8e41c289390506f"
Jul 28 20:30:26.236969 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.236932    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"]
Jul 28 20:30:26.245580 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.245548    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89b18cb8-9c957d86b-62xq7"]
Jul 28 20:30:26.271785 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.267388    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"]
Jul 28 20:30:26.273763 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.273736    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89b18cb8-85d5756b88-chdrj"]
Jul 28 20:30:26.519360 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.519318    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:26.583765 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.583724    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="283528f9-0156-43d2-9675-427f128f1f00" path="/var/lib/kubelet/pods/283528f9-0156-43d2-9675-427f128f1f00/volumes"
Jul 28 20:30:26.584179 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.584159    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="526f6c32-3cb8-41cd-bcb0-59dd47ae97f7" path="/var/lib/kubelet/pods/526f6c32-3cb8-41cd-bcb0-59dd47ae97f7/volumes"
Jul 28 20:30:26.584580 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.584561    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad88daa8-70ac-4f3f-92a7-218e484228a9" path="/var/lib/kubelet/pods/ad88daa8-70ac-4f3f-92a7-218e484228a9/volumes"
Jul 28 20:30:26.584979 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.584962    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff275438-f1d5-48e5-bdcc-0a76eca2d5b0" path="/var/lib/kubelet/pods/ff275438-f1d5-48e5-bdcc-0a76eca2d5b0/volumes"
Jul 28 20:30:26.721732 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.721691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:26.969786 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:26.969697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:27.311649 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.311616    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"]
Jul 28 20:30:27.312069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.311974    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526f6c32-3cb8-41cd-bcb0-59dd47ae97f7" containerName="payload-pre-processing"
Jul 28 20:30:27.312069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.311985    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="526f6c32-3cb8-41cd-bcb0-59dd47ae97f7" containerName="payload-pre-processing"
Jul 28 20:30:27.312069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.311994    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283528f9-0156-43d2-9675-427f128f1f00" containerName="revoke-keys"
Jul 28 20:30:27.312069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.311999    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="283528f9-0156-43d2-9675-427f128f1f00" containerName="revoke-keys"
Jul 28 20:30:27.312069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.312008    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff275438-f1d5-48e5-bdcc-0a76eca2d5b0" containerName="maas-api"
Jul 28 20:30:27.312069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.312014    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff275438-f1d5-48e5-bdcc-0a76eca2d5b0" containerName="maas-api"
Jul 28 20:30:27.312069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.312028    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad88daa8-70ac-4f3f-92a7-218e484228a9" containerName="payload-processing"
Jul 28 20:30:27.312069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.312034    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad88daa8-70ac-4f3f-92a7-218e484228a9" containerName="payload-processing"
Jul 28 20:30:27.312370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.312099    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff275438-f1d5-48e5-bdcc-0a76eca2d5b0" containerName="maas-api"
Jul 28 20:30:27.312370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.312108    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="526f6c32-3cb8-41cd-bcb0-59dd47ae97f7" containerName="payload-pre-processing"
Jul 28 20:30:27.312370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.312114    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad88daa8-70ac-4f3f-92a7-218e484228a9" containerName="payload-processing"
Jul 28 20:30:27.312370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.312120    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f08c12-606b-4c6f-8f80-7b305d36799a" containerName="istio-proxy"
Jul 28 20:30:27.312370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.312127    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="283528f9-0156-43d2-9675-427f128f1f00" containerName="revoke-keys"
Jul 28 20:30:27.316554 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.316532    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.325627 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.325601    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1bd84f55-openshift-default-dockercfg-8jdlj\""
Jul 28 20:30:27.347167 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.347089    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"]
Jul 28 20:30:27.368686 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.368655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:27.450718 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.450684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-certs\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.450718 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.450720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-podinfo\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.450936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.450758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-socket\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.450936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.450785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-data\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.450936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.450821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-credential-socket\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.450936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.450836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-token\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.450936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.450887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istiod-ca-cert\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.450936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.450927    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-envoy\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.451193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.450946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnk8\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-kube-api-access-hvnk8\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-credential-socket\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-token\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552440 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istiod-ca-cert\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552440 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-envoy\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552440 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnk8\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-kube-api-access-hvnk8\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552440 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-certs\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552440 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-podinfo\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552440 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-socket\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-data\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552603    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-credential-socket\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552865 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-data\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.552921 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552903    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-certs\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.553002 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552978    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-socket\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.553052 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.552983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istiod-ca-cert\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.554562 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.554532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-envoy\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.554909 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.554890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-podinfo\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.562445 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.562381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnk8\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-kube-api-access-hvnk8\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.562751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.562722    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-token\") pod \"e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") " pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.626436 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.626399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:27.701197 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.701166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:27.786352 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.786323    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"]
Jul 28 20:30:27.787516 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:30:27.787492    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34cecfb_8826_47bc_bd97_3d30eb76d86d.slice/crio-4ece735f1d1945bd581b2d5407cf6142353842cc2ca10930d7155e890c3fb1e8 WatchSource:0}: Error finding container 4ece735f1d1945bd581b2d5407cf6142353842cc2ca10930d7155e890c3fb1e8: Status 404 returned error can't find the container with id 4ece735f1d1945bd581b2d5407cf6142353842cc2ca10930d7155e890c3fb1e8
Jul 28 20:30:27.789771 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.789728    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:30:27.789850 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.789804    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:30:27.789850 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:27.789838    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:30:28.220739 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:28.220659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" event={"ID":"d34cecfb-8826-47bc-bd97-3d30eb76d86d","Type":"ContainerStarted","Data":"b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41"}
Jul 28 20:30:28.220739 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:28.220708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" event={"ID":"d34cecfb-8826-47bc-bd97-3d30eb76d86d","Type":"ContainerStarted","Data":"4ece735f1d1945bd581b2d5407cf6142353842cc2ca10930d7155e890c3fb1e8"}
Jul 28 20:30:28.249783 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:28.249729    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" podStartSLOduration=1.24968692 podStartE2EDuration="1.24968692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:28.248656218 +0000 UTC m=+1878.102967306" watchObservedRunningTime="2026-07-28 20:30:28.24968692 +0000 UTC m=+1878.103997987"
Jul 28 20:30:28.626765 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:28.626724    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:28.632496 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:28.632459    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:29.225976 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.225939    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:29.227064 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.227041    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:29.632348 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.632315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:29.720715 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.720670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:29.810926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.810891    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"]
Jul 28 20:30:29.814499 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.814476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:29.817168 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.817148    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1bd84f55-serving-cert\""
Jul 28 20:30:29.825817 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.825791    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"]
Jul 28 20:30:29.846768 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.846733    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:29.976434 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.976332    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499218b3-87eb-436d-a069-edd7ea347b76-openshift-service-ca\") pod \"maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:29.976612 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.976510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499218b3-87eb-436d-a069-edd7ea347b76-maas-api-tls\") pod \"maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:29.976612 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:29.976545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h696\" (UniqueName: \"kubernetes.io/projected/499218b3-87eb-436d-a069-edd7ea347b76-kube-api-access-7h696\") pod \"maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:30.077723 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.077689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499218b3-87eb-436d-a069-edd7ea347b76-maas-api-tls\") pod \"maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:30.077942 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.077730    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h696\" (UniqueName: \"kubernetes.io/projected/499218b3-87eb-436d-a069-edd7ea347b76-kube-api-access-7h696\") pod \"maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:30.077942 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.077798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499218b3-87eb-436d-a069-edd7ea347b76-openshift-service-ca\") pod \"maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:30.078504 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.078475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499218b3-87eb-436d-a069-edd7ea347b76-openshift-service-ca\") pod \"maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:30.080147 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.080127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499218b3-87eb-436d-a069-edd7ea347b76-maas-api-tls\") pod \"maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:30.086539 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.086516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h696\" (UniqueName: \"kubernetes.io/projected/499218b3-87eb-436d-a069-edd7ea347b76-kube-api-access-7h696\") pod \"maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:30.126738 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.126699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:30.279019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.278877    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"]
Jul 28 20:30:30.281929 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:30:30.281896    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499218b3_87eb_436d_a069_edd7ea347b76.slice/crio-bf860d767d469018f23171aea18a21f3e1bcd8626d78d0e6f86d7480f7506de9 WatchSource:0}: Error finding container bf860d767d469018f23171aea18a21f3e1bcd8626d78d0e6f86d7480f7506de9: Status 404 returned error can't find the container with id bf860d767d469018f23171aea18a21f3e1bcd8626d78d0e6f86d7480f7506de9
Jul 28 20:30:30.647381 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.647356    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"]
Jul 28 20:30:30.651190 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.651170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:30.654041 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.654020    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1bd84f55-dockercfg-k4z8f\""
Jul 28 20:30:30.654252 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.654237    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1bd84f55\""
Jul 28 20:30:30.659500 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.659479    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"]
Jul 28 20:30:30.727606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.727567    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"]
Jul 28 20:30:30.731789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.731764    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:30.739938 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.739908    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"]
Jul 28 20:30:30.785050 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.785017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmps\" (UniqueName: \"kubernetes.io/projected/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-kube-api-access-dgmps\") pod \"payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552\" (UID: \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:30.785217 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.785166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552\" (UID: \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:30.886010 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.885910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjtr\" (UniqueName: \"kubernetes.io/projected/b3b4f44a-5e58-4c32-b800-a4548693cec4-kube-api-access-rbjtr\") pod \"payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq\" (UID: \"b3b4f44a-5e58-4c32-b800-a4548693cec4\") " pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:30.886010 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.885971    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b4f44a-5e58-4c32-b800-a4548693cec4-plugins-config-volume\") pod \"payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq\" (UID: \"b3b4f44a-5e58-4c32-b800-a4548693cec4\") " pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:30.886199 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.886079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552\" (UID: \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:30.886199 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.886121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmps\" (UniqueName: \"kubernetes.io/projected/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-kube-api-access-dgmps\") pod \"payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552\" (UID: \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:30.886813 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.886795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552\" (UID: \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:30.894118 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.894096    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmps\" (UniqueName: \"kubernetes.io/projected/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-kube-api-access-dgmps\") pod \"payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552\" (UID: \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:30.962645 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.962610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:30.986646 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.986616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjtr\" (UniqueName: \"kubernetes.io/projected/b3b4f44a-5e58-4c32-b800-a4548693cec4-kube-api-access-rbjtr\") pod \"payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq\" (UID: \"b3b4f44a-5e58-4c32-b800-a4548693cec4\") " pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:30.986789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.986675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b4f44a-5e58-4c32-b800-a4548693cec4-plugins-config-volume\") pod \"payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq\" (UID: \"b3b4f44a-5e58-4c32-b800-a4548693cec4\") " pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:30.987658 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.987633    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b4f44a-5e58-4c32-b800-a4548693cec4-plugins-config-volume\") pod \"payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq\" (UID: \"b3b4f44a-5e58-4c32-b800-a4548693cec4\") " pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:30.996253 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:30.996230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjtr\" (UniqueName: \"kubernetes.io/projected/b3b4f44a-5e58-4c32-b800-a4548693cec4-kube-api-access-rbjtr\") pod \"payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq\" (UID: \"b3b4f44a-5e58-4c32-b800-a4548693cec4\") " pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:31.045448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.045412    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:31.104475 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.104426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:31.129333 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.125133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"]
Jul 28 20:30:31.129677 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:30:31.129535    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4aca703_7e1d_4dfd_9536_60bf57ab5c42.slice/crio-8f1ba3cacc5b68f5780b120dc13140c7e34c1c4a856314da4654f2f264e30e4d WatchSource:0}: Error finding container 8f1ba3cacc5b68f5780b120dc13140c7e34c1c4a856314da4654f2f264e30e4d: Status 404 returned error can't find the container with id 8f1ba3cacc5b68f5780b120dc13140c7e34c1c4a856314da4654f2f264e30e4d
Jul 28 20:30:31.207381 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:30:31.207349    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b4f44a_5e58_4c32_b800_a4548693cec4.slice/crio-11f6bc90f6bce3f7e36aa5b11676276ae8fd5f47dacea23c73ab8cdb06c841e9 WatchSource:0}: Error finding container 11f6bc90f6bce3f7e36aa5b11676276ae8fd5f47dacea23c73ab8cdb06c841e9: Status 404 returned error can't find the container with id 11f6bc90f6bce3f7e36aa5b11676276ae8fd5f47dacea23c73ab8cdb06c841e9
Jul 28 20:30:31.207683 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.207664    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"]
Jul 28 20:30:31.210820 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.210794    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:31.240828 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.240793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552" event={"ID":"c4aca703-7e1d-4dfd-9536-60bf57ab5c42","Type":"ContainerStarted","Data":"b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb"}
Jul 28 20:30:31.240999 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.240854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552" event={"ID":"c4aca703-7e1d-4dfd-9536-60bf57ab5c42","Type":"ContainerStarted","Data":"8f1ba3cacc5b68f5780b120dc13140c7e34c1c4a856314da4654f2f264e30e4d"}
Jul 28 20:30:31.240999 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.240882    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:31.242056 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.242028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq" event={"ID":"b3b4f44a-5e58-4c32-b800-a4548693cec4","Type":"ContainerStarted","Data":"11f6bc90f6bce3f7e36aa5b11676276ae8fd5f47dacea23c73ab8cdb06c841e9"}
Jul 28 20:30:31.243350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.243328    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc" event={"ID":"499218b3-87eb-436d-a069-edd7ea347b76","Type":"ContainerStarted","Data":"87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da"}
Jul 28 20:30:31.243436 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.243370    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc" event={"ID":"499218b3-87eb-436d-a069-edd7ea347b76","Type":"ContainerStarted","Data":"bf860d767d469018f23171aea18a21f3e1bcd8626d78d0e6f86d7480f7506de9"}
Jul 28 20:30:31.269037 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.268994    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552" podStartSLOduration=1.268977933 podStartE2EDuration="1.268977933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:31.267532392 +0000 UTC m=+1881.121843458" watchObservedRunningTime="2026-07-28 20:30:31.268977933 +0000 UTC m=+1881.123289009"
Jul 28 20:30:31.296915 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:31.296848    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc" podStartSLOduration=1.986134356 podStartE2EDuration="2.296825706s" totalImagesPullingTime="310.69135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:31.294952325 +0000 UTC m=+1881.149263410" watchObservedRunningTime="2026-07-28 20:30:31.296825706 +0000 UTC m=+1881.151136773"
Jul 28 20:30:32.249704 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:32.249663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq" event={"ID":"b3b4f44a-5e58-4c32-b800-a4548693cec4","Type":"ContainerStarted","Data":"fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95"}
Jul 28 20:30:32.250183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:32.249745    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:32.250183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:32.249942    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:32.266962 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:32.266905    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq" podStartSLOduration=2.266883815 podStartE2EDuration="2.266883815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:32.265813005 +0000 UTC m=+1882.120124075" watchObservedRunningTime="2026-07-28 20:30:32.266883815 +0000 UTC m=+1882.121194882"
Jul 28 20:30:38.260683 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:38.260653    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:30:42.252762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:42.252728    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:30:43.257801 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:43.257772    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:30:44.228090 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:44.227990    2582 patch_prober.go:28] interesting pod/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 20:30:44.228090 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:44.228062    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 20:30:44.315138 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:44.315108    2582 generic.go:356] "Generic (PLEG): container finished" podID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerID="b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41" exitCode=0
Jul 28 20:30:44.315532 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:44.315183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" event={"ID":"d34cecfb-8826-47bc-bd97-3d30eb76d86d","Type":"ContainerDied","Data":"b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41"}
Jul 28 20:30:44.315652 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:44.315636    2582 scope.go:117] "RemoveContainer" containerID="b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41"
Jul 28 20:30:44.316698 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:44.316670    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:30:44.316763 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:44.316750    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:30:44.316805 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:44.316781    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:30:45.321567 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:45.321533    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" event={"ID":"d34cecfb-8826-47bc-bd97-3d30eb76d86d","Type":"ContainerStarted","Data":"11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72"}
Jul 28 20:30:45.627002 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:45.626898    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:45.627282 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:45.627250    2582 patch_prober.go:28] interesting pod/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 20:30:45.627545 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:45.627519    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 20:30:46.627222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:46.627180    2582 patch_prober.go:28] interesting pod/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 20:30:46.627755 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:46.627266    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 20:30:47.627660 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:47.627616    2582 patch_prober.go:28] interesting pod/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 20:30:47.628711 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:47.628251    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 20:30:48.627243 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:48.627192    2582 patch_prober.go:28] interesting pod/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 28 20:30:48.627476 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:48.627267    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 28 20:30:49.631056 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:49.631025    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:49.631591 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:49.631404    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:30:49.632141 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:30:49.632120    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:31:00.658906 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:00.658863    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"]
Jul 28 20:31:00.659399 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:00.659122    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc" podUID="499218b3-87eb-436d-a069-edd7ea347b76" containerName="maas-api" containerID="cri-o://87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da" gracePeriod=30
Jul 28 20:31:00.704752 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:00.704698    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:00.749342 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:00.749304    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"]
Jul 28 20:31:00.749574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:00.749547    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq" podUID="b3b4f44a-5e58-4c32-b800-a4548693cec4" containerName="payload-processing" containerID="cri-o://fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95" gracePeriod=30
Jul 28 20:31:00.773643 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:00.773608    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"]
Jul 28 20:31:00.773866 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:00.773829    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552" podUID="c4aca703-7e1d-4dfd-9536-60bf57ab5c42" containerName="payload-pre-processing" containerID="cri-o://b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb" gracePeriod=30
Jul 28 20:31:01.032044 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.032004    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:31:01.074324 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.074281    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:31:01.079052 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.079018    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:31:01.082604 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.082578    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499218b3-87eb-436d-a069-edd7ea347b76-openshift-service-ca\") pod \"499218b3-87eb-436d-a069-edd7ea347b76\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") "
Jul 28 20:31:01.082743 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.082630    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h696\" (UniqueName: \"kubernetes.io/projected/499218b3-87eb-436d-a069-edd7ea347b76-kube-api-access-7h696\") pod \"499218b3-87eb-436d-a069-edd7ea347b76\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") "
Jul 28 20:31:01.082789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.082753    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499218b3-87eb-436d-a069-edd7ea347b76-maas-api-tls\") pod \"499218b3-87eb-436d-a069-edd7ea347b76\" (UID: \"499218b3-87eb-436d-a069-edd7ea347b76\") "
Jul 28 20:31:01.082907 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.082882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499218b3-87eb-436d-a069-edd7ea347b76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "499218b3-87eb-436d-a069-edd7ea347b76" (UID: "499218b3-87eb-436d-a069-edd7ea347b76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:31:01.083038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.083023    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499218b3-87eb-436d-a069-edd7ea347b76-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:01.084703 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.084677    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499218b3-87eb-436d-a069-edd7ea347b76-kube-api-access-7h696" (OuterVolumeSpecName: "kube-api-access-7h696") pod "499218b3-87eb-436d-a069-edd7ea347b76" (UID: "499218b3-87eb-436d-a069-edd7ea347b76"). InnerVolumeSpecName "kube-api-access-7h696". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:31:01.084809 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.084734    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499218b3-87eb-436d-a069-edd7ea347b76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "499218b3-87eb-436d-a069-edd7ea347b76" (UID: "499218b3-87eb-436d-a069-edd7ea347b76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:31:01.157087 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.157048    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"]
Jul 28 20:31:01.157369 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.157339    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy" containerID="cri-o://11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72" gracePeriod=30
Jul 28 20:31:01.183572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.183490    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-plugins-config-volume\") pod \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\" (UID: \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\") "
Jul 28 20:31:01.183572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.183535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbjtr\" (UniqueName: \"kubernetes.io/projected/b3b4f44a-5e58-4c32-b800-a4548693cec4-kube-api-access-rbjtr\") pod \"b3b4f44a-5e58-4c32-b800-a4548693cec4\" (UID: \"b3b4f44a-5e58-4c32-b800-a4548693cec4\") "
Jul 28 20:31:01.183808 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.183633    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgmps\" (UniqueName: \"kubernetes.io/projected/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-kube-api-access-dgmps\") pod \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\" (UID: \"c4aca703-7e1d-4dfd-9536-60bf57ab5c42\") "
Jul 28 20:31:01.183808 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.183649    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b4f44a-5e58-4c32-b800-a4548693cec4-plugins-config-volume\") pod \"b3b4f44a-5e58-4c32-b800-a4548693cec4\" (UID: \"b3b4f44a-5e58-4c32-b800-a4548693cec4\") "
Jul 28 20:31:01.183925 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.183883    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4aca703-7e1d-4dfd-9536-60bf57ab5c42" (UID: "c4aca703-7e1d-4dfd-9536-60bf57ab5c42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:31:01.183925 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.183894    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499218b3-87eb-436d-a069-edd7ea347b76-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:01.183925 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.183915    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h696\" (UniqueName: \"kubernetes.io/projected/499218b3-87eb-436d-a069-edd7ea347b76-kube-api-access-7h696\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:01.184052 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.183997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b4f44a-5e58-4c32-b800-a4548693cec4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3b4f44a-5e58-4c32-b800-a4548693cec4" (UID: "b3b4f44a-5e58-4c32-b800-a4548693cec4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:31:01.185793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.185768    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-kube-api-access-dgmps" (OuterVolumeSpecName: "kube-api-access-dgmps") pod "c4aca703-7e1d-4dfd-9536-60bf57ab5c42" (UID: "c4aca703-7e1d-4dfd-9536-60bf57ab5c42"). InnerVolumeSpecName "kube-api-access-dgmps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:31:01.185900 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.185840    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b4f44a-5e58-4c32-b800-a4548693cec4-kube-api-access-rbjtr" (OuterVolumeSpecName: "kube-api-access-rbjtr") pod "b3b4f44a-5e58-4c32-b800-a4548693cec4" (UID: "b3b4f44a-5e58-4c32-b800-a4548693cec4"). InnerVolumeSpecName "kube-api-access-rbjtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:31:01.284793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.284758    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbjtr\" (UniqueName: \"kubernetes.io/projected/b3b4f44a-5e58-4c32-b800-a4548693cec4-kube-api-access-rbjtr\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:01.284793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.284789    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmps\" (UniqueName: \"kubernetes.io/projected/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-kube-api-access-dgmps\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:01.284793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.284798    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b4f44a-5e58-4c32-b800-a4548693cec4-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:01.284793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.284807    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aca703-7e1d-4dfd-9536-60bf57ab5c42-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:01.413462 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.413424    2582 generic.go:356] "Generic (PLEG): container finished" podID="b3b4f44a-5e58-4c32-b800-a4548693cec4" containerID="fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95" exitCode=0
Jul 28 20:31:01.413656 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.413476    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"
Jul 28 20:31:01.413656 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.413506    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq" event={"ID":"b3b4f44a-5e58-4c32-b800-a4548693cec4","Type":"ContainerDied","Data":"fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95"}
Jul 28 20:31:01.413656 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.413567    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq" event={"ID":"b3b4f44a-5e58-4c32-b800-a4548693cec4","Type":"ContainerDied","Data":"11f6bc90f6bce3f7e36aa5b11676276ae8fd5f47dacea23c73ab8cdb06c841e9"}
Jul 28 20:31:01.413656 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.413597    2582 scope.go:117] "RemoveContainer" containerID="fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95"
Jul 28 20:31:01.414930 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.414894    2582 generic.go:356] "Generic (PLEG): container finished" podID="499218b3-87eb-436d-a069-edd7ea347b76" containerID="87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da" exitCode=0
Jul 28 20:31:01.415005 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.414936    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"
Jul 28 20:31:01.415005 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.414978    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc" event={"ID":"499218b3-87eb-436d-a069-edd7ea347b76","Type":"ContainerDied","Data":"87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da"}
Jul 28 20:31:01.415093 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.415024    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc" event={"ID":"499218b3-87eb-436d-a069-edd7ea347b76","Type":"ContainerDied","Data":"bf860d767d469018f23171aea18a21f3e1bcd8626d78d0e6f86d7480f7506de9"}
Jul 28 20:31:01.416455 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.416432    2582 generic.go:356] "Generic (PLEG): container finished" podID="c4aca703-7e1d-4dfd-9536-60bf57ab5c42" containerID="b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb" exitCode=0
Jul 28 20:31:01.416588 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.416468    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"
Jul 28 20:31:01.416588 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.416491    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552" event={"ID":"c4aca703-7e1d-4dfd-9536-60bf57ab5c42","Type":"ContainerDied","Data":"b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb"}
Jul 28 20:31:01.416588 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.416528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552" event={"ID":"c4aca703-7e1d-4dfd-9536-60bf57ab5c42","Type":"ContainerDied","Data":"8f1ba3cacc5b68f5780b120dc13140c7e34c1c4a856314da4654f2f264e30e4d"}
Jul 28 20:31:01.425847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.425832    2582 scope.go:117] "RemoveContainer" containerID="fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95"
Jul 28 20:31:01.426210 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:31:01.426162    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95\": container with ID starting with fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95 not found: ID does not exist" containerID="fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95"
Jul 28 20:31:01.426332 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.426225    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95"} err="failed to get container status \"fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95\": rpc error: code = NotFound desc = could not find container \"fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95\": container with ID starting with fb692cd08cc5d289e7727e0db6cba75233e5248b20110b7b4048f2545fd37e95 not found: ID does not exist"
Jul 28 20:31:01.426332 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.426249    2582 scope.go:117] "RemoveContainer" containerID="87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da"
Jul 28 20:31:01.437121 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.437102    2582 scope.go:117] "RemoveContainer" containerID="87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da"
Jul 28 20:31:01.437459 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:31:01.437431    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da\": container with ID starting with 87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da not found: ID does not exist" containerID="87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da"
Jul 28 20:31:01.437565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.437461    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da"} err="failed to get container status \"87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da\": rpc error: code = NotFound desc = could not find container \"87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da\": container with ID starting with 87e8b823be35778a2eccac530f97d5ab3e20ccc0aad3caef0fae25676a3e36da not found: ID does not exist"
Jul 28 20:31:01.437565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.437480    2582 scope.go:117] "RemoveContainer" containerID="b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb"
Jul 28 20:31:01.443600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.443576    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"]
Jul 28 20:31:01.445690 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.445673    2582 scope.go:117] "RemoveContainer" containerID="b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb"
Jul 28 20:31:01.445979 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:31:01.445961    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb\": container with ID starting with b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb not found: ID does not exist" containerID="b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb"
Jul 28 20:31:01.446040 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.445990    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb"} err="failed to get container status \"b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb\": rpc error: code = NotFound desc = could not find container \"b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb\": container with ID starting with b7ace8295cfe2e2c62e0d9c0c022356c378161ae2904f54d676764d77f569bdb not found: ID does not exist"
Jul 28 20:31:01.450074 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.450054    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1bd84f55-5f78fc9868-55552"]
Jul 28 20:31:01.465049 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.465020    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"]
Jul 28 20:31:01.473818 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.473793    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1bd84f55-744bbdccf4-7cldq"]
Jul 28 20:31:01.494612 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.494582    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"]
Jul 28 20:31:01.501535 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:01.501509    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1bd84f55-867f9bf8d5-vmnsc"]
Jul 28 20:31:02.583071 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:02.583036    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="499218b3-87eb-436d-a069-edd7ea347b76" path="/var/lib/kubelet/pods/499218b3-87eb-436d-a069-edd7ea347b76/volumes"
Jul 28 20:31:02.583471 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:02.583428    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3b4f44a-5e58-4c32-b800-a4548693cec4" path="/var/lib/kubelet/pods/b3b4f44a-5e58-4c32-b800-a4548693cec4/volumes"
Jul 28 20:31:02.583754 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:02.583741    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4aca703-7e1d-4dfd-9536-60bf57ab5c42" path="/var/lib/kubelet/pods/c4aca703-7e1d-4dfd-9536-60bf57ab5c42/volumes"
Jul 28 20:31:02.753852 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:02.753811    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:03.528848 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:03.528807    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:04.633441 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:04.633403    2582 patch_prober.go:28] interesting pod/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:31:04.633823 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:04.633482    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:31:06.413705 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.413676    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:31:06.454431 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.454320    2582 generic.go:356] "Generic (PLEG): container finished" podID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerID="11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72" exitCode=0
Jul 28 20:31:06.454608 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.454562    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"
Jul 28 20:31:06.454887 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.454852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" event={"ID":"d34cecfb-8826-47bc-bd97-3d30eb76d86d","Type":"ContainerDied","Data":"11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72"}
Jul 28 20:31:06.454973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.454935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5" event={"ID":"d34cecfb-8826-47bc-bd97-3d30eb76d86d","Type":"ContainerDied","Data":"4ece735f1d1945bd581b2d5407cf6142353842cc2ca10930d7155e890c3fb1e8"}
Jul 28 20:31:06.455020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.454982    2582 scope.go:117] "RemoveContainer" containerID="11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72"
Jul 28 20:31:06.465613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.465592    2582 scope.go:117] "RemoveContainer" containerID="b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41"
Jul 28 20:31:06.475214 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.475189    2582 scope.go:117] "RemoveContainer" containerID="11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72"
Jul 28 20:31:06.475628 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:31:06.475602    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72\": container with ID starting with 11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72 not found: ID does not exist" containerID="11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72"
Jul 28 20:31:06.475751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.475638    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72"} err="failed to get container status \"11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72\": rpc error: code = NotFound desc = could not find container \"11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72\": container with ID starting with 11256f5d88421df304ae61cabb9886840f7c85d13eec5bae4432fd92bd11ce72 not found: ID does not exist"
Jul 28 20:31:06.475751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.475662    2582 scope.go:117] "RemoveContainer" containerID="b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41"
Jul 28 20:31:06.475982 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:31:06.475960    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41\": container with ID starting with b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41 not found: ID does not exist" containerID="b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41"
Jul 28 20:31:06.476055 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.475993    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41"} err="failed to get container status \"b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41\": rpc error: code = NotFound desc = could not find container \"b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41\": container with ID starting with b11ce29d29f3ca1363959831c0b5b7193b5dc6a226c09d83be56110a1a58dd41 not found: ID does not exist"
Jul 28 20:31:06.540263 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540221    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvnk8\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-kube-api-access-hvnk8\") pod \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") "
Jul 28 20:31:06.540469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540272    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-podinfo\") pod \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") "
Jul 28 20:31:06.540469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540318    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istiod-ca-cert\") pod \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") "
Jul 28 20:31:06.540469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540347    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-envoy\") pod \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") "
Jul 28 20:31:06.540469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540459    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-token\") pod \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") "
Jul 28 20:31:06.540723 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540583    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-data\") pod \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") "
Jul 28 20:31:06.540723 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-certs\") pod \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") "
Jul 28 20:31:06.540910 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d34cecfb-8826-47bc-bd97-3d30eb76d86d" (UID: "d34cecfb-8826-47bc-bd97-3d30eb76d86d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:31:06.541186 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540942    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-credential-socket\") pod \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") "
Jul 28 20:31:06.541186 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.540976    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-socket\") pod \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\" (UID: \"d34cecfb-8826-47bc-bd97-3d30eb76d86d\") "
Jul 28 20:31:06.541323 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.541313    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istiod-ca-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:06.541390 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.541363    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d34cecfb-8826-47bc-bd97-3d30eb76d86d" (UID: "d34cecfb-8826-47bc-bd97-3d30eb76d86d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:31:06.541500 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.541480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d34cecfb-8826-47bc-bd97-3d30eb76d86d" (UID: "d34cecfb-8826-47bc-bd97-3d30eb76d86d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:31:06.542236 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.542210    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d34cecfb-8826-47bc-bd97-3d30eb76d86d" (UID: "d34cecfb-8826-47bc-bd97-3d30eb76d86d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:31:06.542411 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.542239    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d34cecfb-8826-47bc-bd97-3d30eb76d86d" (UID: "d34cecfb-8826-47bc-bd97-3d30eb76d86d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:31:06.543680 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.543645    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-kube-api-access-hvnk8" (OuterVolumeSpecName: "kube-api-access-hvnk8") pod "d34cecfb-8826-47bc-bd97-3d30eb76d86d" (UID: "d34cecfb-8826-47bc-bd97-3d30eb76d86d"). InnerVolumeSpecName "kube-api-access-hvnk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:31:06.544014 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.543991    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d34cecfb-8826-47bc-bd97-3d30eb76d86d" (UID: "d34cecfb-8826-47bc-bd97-3d30eb76d86d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:31:06.544210 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.544193    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d34cecfb-8826-47bc-bd97-3d30eb76d86d" (UID: "d34cecfb-8826-47bc-bd97-3d30eb76d86d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:31:06.544603 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.544574    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d34cecfb-8826-47bc-bd97-3d30eb76d86d" (UID: "d34cecfb-8826-47bc-bd97-3d30eb76d86d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:31:06.642474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.642430    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-podinfo\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:06.642474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.642465    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-envoy\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:06.642474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.642478    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:06.642474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.642486    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-istio-data\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:06.642859 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.642495    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:06.642859 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.642503    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-credential-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:06.642859 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.642513    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34cecfb-8826-47bc-bd97-3d30eb76d86d-workload-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:06.642859 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.642522    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvnk8\" (UniqueName: \"kubernetes.io/projected/d34cecfb-8826-47bc-bd97-3d30eb76d86d-kube-api-access-hvnk8\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:31:06.776823 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.776791    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"]
Jul 28 20:31:06.780076 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:06.780051    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1bd84f55-openshift-default-5576dcfd44-p9rd5"]
Jul 28 20:31:08.582895 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:08.582860    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" path="/var/lib/kubelet/pods/d34cecfb-8826-47bc-bd97-3d30eb76d86d/volumes"
Jul 28 20:31:09.815097 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:09.815062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:09.967730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:09.967680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:10.086155 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:10.086052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:10.207544 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:10.207506    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:10.620579 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:10.620538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:36.538948 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:36.538911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:43.278438 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:43.278395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:43.380540 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:43.380503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:43.499006 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:43.498968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:59.372923 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:31:59.372887    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:05.740130 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:05.740081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:05.845350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:05.845314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:05.972879 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:05.972849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:16.487508 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:16.487449    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:17.299328 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:17.299274    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:25.385535 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:25.385496    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:25.589456 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:25.589419    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:25.817750 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:25.817708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:39.384033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:39.383972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:39.484876 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:39.484840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:39.591471 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:39.591428    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:41.372025 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:41.371986    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:48.145139 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:48.145095    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:48.248375 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:48.248339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:48.882960 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:48.882921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:55.805195 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:55.805158    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:55.984123 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:55.984090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:56.125000 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.124892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:56.472192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.472092    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"]
Jul 28 20:32:56.473224 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.473201    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4aca703-7e1d-4dfd-9536-60bf57ab5c42" containerName="payload-pre-processing"
Jul 28 20:32:56.473396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.473384    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4aca703-7e1d-4dfd-9536-60bf57ab5c42" containerName="payload-pre-processing"
Jul 28 20:32:56.473498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.473489    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b4f44a-5e58-4c32-b800-a4548693cec4" containerName="payload-processing"
Jul 28 20:32:56.473575 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.473564    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b4f44a-5e58-4c32-b800-a4548693cec4" containerName="payload-processing"
Jul 28 20:32:56.473690 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.473680    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy"
Jul 28 20:32:56.473762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.473754    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy"
Jul 28 20:32:56.473844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.473835    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="499218b3-87eb-436d-a069-edd7ea347b76" containerName="maas-api"
Jul 28 20:32:56.473914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.473906    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="499218b3-87eb-436d-a069-edd7ea347b76" containerName="maas-api"
Jul 28 20:32:56.474003 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.473994    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy"
Jul 28 20:32:56.474074 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.474066    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy"
Jul 28 20:32:56.474350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.474337    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy"
Jul 28 20:32:56.474447 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.474437    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4aca703-7e1d-4dfd-9536-60bf57ab5c42" containerName="payload-pre-processing"
Jul 28 20:32:56.474537 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.474528    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b4f44a-5e58-4c32-b800-a4548693cec4" containerName="payload-processing"
Jul 28 20:32:56.474613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.474604    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="499218b3-87eb-436d-a069-edd7ea347b76" containerName="maas-api"
Jul 28 20:32:56.478838 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.478815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.487886 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.487869    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e9e0fd3a-openshift-default-dockercfg-b2p99\""
Jul 28 20:32:56.554955 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.554917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:56.558652 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.558625    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"]
Jul 28 20:32:56.631266 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.631234    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-envoy\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.631464 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.631277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istiod-ca-cert\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.631464 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.631334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7gzv\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-kube-api-access-x7gzv\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.631464 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.631363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-podinfo\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.631464 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.631388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-data\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.631651 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.631493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-credential-socket\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.631651 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.631566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-socket\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.631651 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.631593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-token\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.631651 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.631639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-certs\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732353 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-certs\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-envoy\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istiod-ca-cert\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7gzv\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-kube-api-access-x7gzv\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-podinfo\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-data\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-credential-socket\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-socket\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-token\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.732877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.732750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-certs\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.733047 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.733007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-data\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.733133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.733105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-credential-socket\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.733191 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.733127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istiod-ca-cert\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.733191 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.733174    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-socket\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.734735 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.734715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-envoy\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.735148 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.735123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-podinfo\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.743749 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.743721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7gzv\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-kube-api-access-x7gzv\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.755121 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.755095    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-token\") pod \"e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") " pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.789245 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.789215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:56.851054 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.851017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:56.977239 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.977215    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"]
Jul 28 20:32:56.979484 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:32:56.979462    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca158e2e_5ec3_406b_a326_ad68b395c5b9.slice/crio-88733fb56924ed9b166730917d443105c90fee0810c7dbbe042a6c4fbb678265 WatchSource:0}: Error finding container 88733fb56924ed9b166730917d443105c90fee0810c7dbbe042a6c4fbb678265: Status 404 returned error can't find the container with id 88733fb56924ed9b166730917d443105c90fee0810c7dbbe042a6c4fbb678265
Jul 28 20:32:56.981624 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.981590    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:32:56.981731 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.981653    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:32:56.981731 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.981682    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:32:56.985923 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:56.985893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" event={"ID":"ca158e2e-5ec3-406b-a326-ad68b395c5b9","Type":"ContainerStarted","Data":"88733fb56924ed9b166730917d443105c90fee0810c7dbbe042a6c4fbb678265"}
Jul 28 20:32:57.993058 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:57.993021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" event={"ID":"ca158e2e-5ec3-406b-a326-ad68b395c5b9","Type":"ContainerStarted","Data":"f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7"}
Jul 28 20:32:58.023929 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:58.023875    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" podStartSLOduration=2.023858138 podStartE2EDuration="2.023858138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:32:58.022472703 +0000 UTC m=+2027.876783771" watchObservedRunningTime="2026-07-28 20:32:58.023858138 +0000 UTC m=+2027.878169204"
Jul 28 20:32:58.789662 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:58.789628    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:58.794445 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:58.794422    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:58.997942 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:58.997910    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:32:58.998951 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:32:58.998932    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:33:00.076007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.075950    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:00.173656 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.173623    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"]
Jul 28 20:33:00.174330 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.174286    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d34cecfb-8826-47bc-bd97-3d30eb76d86d" containerName="istio-proxy"
Jul 28 20:33:00.176856 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.176827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:00.177938 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.177905    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.182183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.182161    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e9e0fd3a-serving-cert\""
Jul 28 20:33:00.195080 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.195053    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"]
Jul 28 20:33:00.268146 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.268107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-maas-api-tls\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.268374 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.268187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-openshift-service-ca\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.268374 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.268220    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd966\" (UniqueName: \"kubernetes.io/projected/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-kube-api-access-bd966\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.285757 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.285701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:00.368941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.368846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-openshift-service-ca\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.368941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.368892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd966\" (UniqueName: \"kubernetes.io/projected/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-kube-api-access-bd966\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.369151 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.368972    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-maas-api-tls\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.369151 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:33:00.369127    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-serving-cert: secret "maas-api-e2e-mt-e9e0fd3a-serving-cert" not found
Jul 28 20:33:00.369225 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:33:00.369205    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-maas-api-tls podName:cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99 nodeName:}" failed. No retries permitted until 2026-07-28 20:33:00.869179425 +0000 UTC m=+2030.723490492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-maas-api-tls") pod "maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh" (UID: "cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99") : secret "maas-api-e2e-mt-e9e0fd3a-serving-cert" not found
Jul 28 20:33:00.369660 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.369639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-openshift-service-ca\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.387686 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.387651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd966\" (UniqueName: \"kubernetes.io/projected/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-kube-api-access-bd966\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.805926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.805888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:00.873865 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.873826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-maas-api-tls\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.876403 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.876383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-maas-api-tls\") pod \"maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:00.955586 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:00.955549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:01.060843 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.060749    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"]
Jul 28 20:33:01.064746 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.064718    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:01.068131 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.068106    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e9e0fd3a-dockercfg-skbcg\""
Jul 28 20:33:01.068526 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.068504    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e9e0fd3a\""
Jul 28 20:33:01.075755 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.075725    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"]
Jul 28 20:33:01.091066 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.091037    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:01.176265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.176227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-plugins-config-volume\") pod \"payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv\" (UID: \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\") " pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:01.176492 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.176273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8km\" (UniqueName: \"kubernetes.io/projected/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-kube-api-access-rn8km\") pod \"payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv\" (UID: \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\") " pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:01.251964 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.251936    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"]
Jul 28 20:33:01.253143 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:33:01.253116    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafe7fec_c0b9_48d0_8c3e_cb492f6f1d99.slice/crio-05290b05910049ad8b04bf2adf65923a0689f82e698e45801b3dfeab0a142930 WatchSource:0}: Error finding container 05290b05910049ad8b04bf2adf65923a0689f82e698e45801b3dfeab0a142930: Status 404 returned error can't find the container with id 05290b05910049ad8b04bf2adf65923a0689f82e698e45801b3dfeab0a142930
Jul 28 20:33:01.277070 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.277040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-plugins-config-volume\") pod \"payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv\" (UID: \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\") " pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:01.277236 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.277079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8km\" (UniqueName: \"kubernetes.io/projected/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-kube-api-access-rn8km\") pod \"payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv\" (UID: \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\") " pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:01.277746 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.277710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-plugins-config-volume\") pod \"payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv\" (UID: \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\") " pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:01.287230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.287204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8km\" (UniqueName: \"kubernetes.io/projected/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-kube-api-access-rn8km\") pod \"payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv\" (UID: \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\") " pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:01.376908 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.376819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:01.750082 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.750051    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"]
Jul 28 20:33:01.752121 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:33:01.752084    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d64be69_4ca5_4d8d_bba6_fff0c4769db1.slice/crio-77835981ab8a1d6376a905272af0d2e6f74c97b8e7e7512498c63ac0129d2afb WatchSource:0}: Error finding container 77835981ab8a1d6376a905272af0d2e6f74c97b8e7e7512498c63ac0129d2afb: Status 404 returned error can't find the container with id 77835981ab8a1d6376a905272af0d2e6f74c97b8e7e7512498c63ac0129d2afb
Jul 28 20:33:01.959609 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:01.959581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:02.015753 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:02.015659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh" event={"ID":"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99","Type":"ContainerStarted","Data":"4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1"}
Jul 28 20:33:02.015753 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:02.015718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:02.015753 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:02.015743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh" event={"ID":"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99","Type":"ContainerStarted","Data":"05290b05910049ad8b04bf2adf65923a0689f82e698e45801b3dfeab0a142930"}
Jul 28 20:33:02.017154 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:02.017127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv" event={"ID":"0d64be69-4ca5-4d8d-bba6-fff0c4769db1","Type":"ContainerStarted","Data":"6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f"}
Jul 28 20:33:02.017264 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:02.017167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv" event={"ID":"0d64be69-4ca5-4d8d-bba6-fff0c4769db1","Type":"ContainerStarted","Data":"77835981ab8a1d6376a905272af0d2e6f74c97b8e7e7512498c63ac0129d2afb"}
Jul 28 20:33:02.017264 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:02.017208    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:02.118017 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:02.117966    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh" podStartSLOduration=1.46330716 podStartE2EDuration="2.117948347s" totalImagesPullingTime="654.641187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:33:02.073385577 +0000 UTC m=+2031.927696644" watchObservedRunningTime="2026-07-28 20:33:02.117948347 +0000 UTC m=+2031.972259468"
Jul 28 20:33:02.119790 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:02.119753    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv" podStartSLOduration=1.119743483 podStartE2EDuration="1.119743483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:33:02.114901155 +0000 UTC m=+2031.969212221" watchObservedRunningTime="2026-07-28 20:33:02.119743483 +0000 UTC m=+2031.974054550"
Jul 28 20:33:08.034353 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:08.033627    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:13.025692 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:13.025660    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:13.084230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:13.084140    2582 generic.go:356] "Generic (PLEG): container finished" podID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerID="f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7" exitCode=0
Jul 28 20:33:13.084230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:13.084210    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" event={"ID":"ca158e2e-5ec3-406b-a326-ad68b395c5b9","Type":"ContainerDied","Data":"f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7"}
Jul 28 20:33:13.084692 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:13.084675    2582 scope.go:117] "RemoveContainer" containerID="f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7"
Jul 28 20:33:13.085737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:13.085705    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:33:13.085815 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:13.085799    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:33:13.085853 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:13.085837    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:33:13.998828 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:13.998777    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:33:14.091072 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:14.091031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" event={"ID":"ca158e2e-5ec3-406b-a326-ad68b395c5b9","Type":"ContainerStarted","Data":"7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2"}
Jul 28 20:33:14.790446 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:14.790408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:33:14.791600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:14.791570    2582 patch_prober.go:28] interesting pod/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 28 20:33:14.791725 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:14.791622    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 28 20:33:15.790529 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:15.790492    2582 patch_prober.go:28] interesting pod/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 28 20:33:15.790971 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:15.790546    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 28 20:33:16.790118 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:16.790075    2582 patch_prober.go:28] interesting pod/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 28 20:33:16.790333 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:16.790136    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 28 20:33:17.789867 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:17.789827    2582 patch_prober.go:28] interesting pod/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 28 20:33:17.790279 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:17.789890    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 28 20:33:18.793674 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:18.793642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:33:18.794161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:18.793918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:33:18.794581 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:18.794563    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:33:19.950689 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:19.950655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:24.958005 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:24.957966    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:25.604733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:25.604699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:25.705035 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:25.704998    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:26.064972 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:26.064928    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:31.842663 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:31.842632    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:31.906942 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:31.906895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:42.311735 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:42.311628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:42.372269 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:42.372228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:42.757957 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:42.757919    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"]
Jul 28 20:33:42.758206 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:42.758179    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh" podUID="cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" containerName="maas-api" containerID="cri-o://4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1" gracePeriod=30
Jul 28 20:33:42.800856 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:42.800824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:42.818149 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:42.818116    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"]
Jul 28 20:33:42.818374 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:42.818351    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv" podUID="0d64be69-4ca5-4d8d-bba6-fff0c4769db1" containerName="payload-processing" containerID="cri-o://6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f" gracePeriod=30
Jul 28 20:33:42.904493 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:42.904442    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:43.002561 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.002365    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:43.025505 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.025403    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv" podUID="0d64be69-4ca5-4d8d-bba6-fff0c4769db1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.70:9004: connect: connection refused"
Jul 28 20:33:43.078952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.078922    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:43.086052 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.086029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-maas-api-tls\") pod \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") "
Jul 28 20:33:43.086161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.086060    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd966\" (UniqueName: \"kubernetes.io/projected/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-kube-api-access-bd966\") pod \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") "
Jul 28 20:33:43.086161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.086156    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-openshift-service-ca\") pod \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\" (UID: \"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99\") "
Jul 28 20:33:43.086553 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.086526    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" (UID: "cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:33:43.088254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.088228    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" (UID: "cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:33:43.088254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.088247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-kube-api-access-bd966" (OuterVolumeSpecName: "kube-api-access-bd966") pod "cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" (UID: "cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99"). InnerVolumeSpecName "kube-api-access-bd966". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:33:43.102647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.102602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:43.106286 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.106261    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:43.186595 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.186557    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-plugins-config-volume\") pod \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\" (UID: \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\") "
Jul 28 20:33:43.186752 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.186639    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn8km\" (UniqueName: \"kubernetes.io/projected/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-kube-api-access-rn8km\") pod \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\" (UID: \"0d64be69-4ca5-4d8d-bba6-fff0c4769db1\") "
Jul 28 20:33:43.186849 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.186837    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:43.186887 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.186851    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:43.186887 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.186861    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd966\" (UniqueName: \"kubernetes.io/projected/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99-kube-api-access-bd966\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:43.187029 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.187004    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d64be69-4ca5-4d8d-bba6-fff0c4769db1" (UID: "0d64be69-4ca5-4d8d-bba6-fff0c4769db1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:33:43.188740 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.188719    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-kube-api-access-rn8km" (OuterVolumeSpecName: "kube-api-access-rn8km") pod "0d64be69-4ca5-4d8d-bba6-fff0c4769db1" (UID: "0d64be69-4ca5-4d8d-bba6-fff0c4769db1"). InnerVolumeSpecName "kube-api-access-rn8km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:33:43.232501 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.232464    2582 generic.go:356] "Generic (PLEG): container finished" podID="cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" containerID="4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1" exitCode=0
Jul 28 20:33:43.232695 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.232507    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"
Jul 28 20:33:43.232695 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.232544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh" event={"ID":"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99","Type":"ContainerDied","Data":"4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1"}
Jul 28 20:33:43.232695 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.232605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh" event={"ID":"cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99","Type":"ContainerDied","Data":"05290b05910049ad8b04bf2adf65923a0689f82e698e45801b3dfeab0a142930"}
Jul 28 20:33:43.232695 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.232638    2582 scope.go:117] "RemoveContainer" containerID="4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1"
Jul 28 20:33:43.234164 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.234132    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d64be69-4ca5-4d8d-bba6-fff0c4769db1" containerID="6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f" exitCode=0
Jul 28 20:33:43.234316 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.234157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv" event={"ID":"0d64be69-4ca5-4d8d-bba6-fff0c4769db1","Type":"ContainerDied","Data":"6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f"}
Jul 28 20:33:43.234316 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.234176    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"
Jul 28 20:33:43.234316 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.234208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv" event={"ID":"0d64be69-4ca5-4d8d-bba6-fff0c4769db1","Type":"ContainerDied","Data":"77835981ab8a1d6376a905272af0d2e6f74c97b8e7e7512498c63ac0129d2afb"}
Jul 28 20:33:43.243108 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.243048    2582 scope.go:117] "RemoveContainer" containerID="4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1"
Jul 28 20:33:43.243439 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:33:43.243419    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1\": container with ID starting with 4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1 not found: ID does not exist" containerID="4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1"
Jul 28 20:33:43.243529 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.243445    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1"} err="failed to get container status \"4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1\": rpc error: code = NotFound desc = could not find container \"4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1\": container with ID starting with 4e96f68e0343a3c4fbcc7420794991a82fe81830cc6df107adcfff500ebc3eb1 not found: ID does not exist"
Jul 28 20:33:43.243529 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.243462    2582 scope.go:117] "RemoveContainer" containerID="6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f"
Jul 28 20:33:43.251529 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.251513    2582 scope.go:117] "RemoveContainer" containerID="6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f"
Jul 28 20:33:43.251825 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:33:43.251804    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f\": container with ID starting with 6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f not found: ID does not exist" containerID="6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f"
Jul 28 20:33:43.251877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.251833    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f"} err="failed to get container status \"6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f\": rpc error: code = NotFound desc = could not find container \"6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f\": container with ID starting with 6dfffd2e51d5edb50959b19ae229f135dd93237682cbc0a536caa9ae2ba0a19f not found: ID does not exist"
Jul 28 20:33:43.259614 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.259593    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"]
Jul 28 20:33:43.267799 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.267777    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh"]
Jul 28 20:33:43.291028 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.290916    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"]
Jul 28 20:33:43.291656 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.291634    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:43.291767 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.291659    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8km\" (UniqueName: \"kubernetes.io/projected/0d64be69-4ca5-4d8d-bba6-fff0c4769db1-kube-api-access-rn8km\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:43.293511 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.293489    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e9e0fd3a-7ccb79444b-pnkhv"]
Jul 28 20:33:43.393156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:43.393114    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:44.582490 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:44.582462    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d64be69-4ca5-4d8d-bba6-fff0c4769db1" path="/var/lib/kubelet/pods/0d64be69-4ca5-4d8d-bba6-fff0c4769db1/volumes"
Jul 28 20:33:44.582877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:44.582852    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" path="/var/lib/kubelet/pods/cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99/volumes"
Jul 28 20:33:46.026222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:46.026177    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9e0fd3a-7d5bc47997-blzjh" podUID="cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.69:8443/health\": context deadline exceeded"
Jul 28 20:33:49.231444 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:49.231402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:49.297772 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:49.297741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:49.467945 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:49.467911    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"]
Jul 28 20:33:49.468131 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:49.468108    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy" containerID="cri-o://7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2" gracePeriod=30
Jul 28 20:33:49.496380 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:49.496338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:49.695104 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:49.695064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:49.831224 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:49.831104    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:49.937562 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:49.937507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:50.054988 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:50.054954    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:54.730126 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.730063    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:33:54.774634 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.774599    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-credential-socket\") pod \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") "
Jul 28 20:33:54.774634 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.774636    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-socket\") pod \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") "
Jul 28 20:33:54.774891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.774665    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-certs\") pod \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") "
Jul 28 20:33:54.774891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.774701    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7gzv\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-kube-api-access-x7gzv\") pod \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") "
Jul 28 20:33:54.774891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.774724    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istiod-ca-cert\") pod \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") "
Jul 28 20:33:54.774891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.774752    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-token\") pod \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") "
Jul 28 20:33:54.775115 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.774915    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-data\") pod \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") "
Jul 28 20:33:54.775115 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.774971    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-podinfo\") pod \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") "
Jul 28 20:33:54.775115 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.774913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca158e2e-5ec3-406b-a326-ad68b395c5b9" (UID: "ca158e2e-5ec3-406b-a326-ad68b395c5b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:33:54.775115 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.775016    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-envoy\") pod \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\" (UID: \"ca158e2e-5ec3-406b-a326-ad68b395c5b9\") "
Jul 28 20:33:54.775362 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.775263    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca158e2e-5ec3-406b-a326-ad68b395c5b9" (UID: "ca158e2e-5ec3-406b-a326-ad68b395c5b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:33:54.775362 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.775309    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:54.775482 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.775459    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca158e2e-5ec3-406b-a326-ad68b395c5b9" (UID: "ca158e2e-5ec3-406b-a326-ad68b395c5b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:33:54.775536 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.775513    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca158e2e-5ec3-406b-a326-ad68b395c5b9" (UID: "ca158e2e-5ec3-406b-a326-ad68b395c5b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:33:54.775788 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.775763    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca158e2e-5ec3-406b-a326-ad68b395c5b9" (UID: "ca158e2e-5ec3-406b-a326-ad68b395c5b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:33:54.777320 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.777275    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca158e2e-5ec3-406b-a326-ad68b395c5b9" (UID: "ca158e2e-5ec3-406b-a326-ad68b395c5b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:33:54.777432 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.777279    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-kube-api-access-x7gzv" (OuterVolumeSpecName: "kube-api-access-x7gzv") pod "ca158e2e-5ec3-406b-a326-ad68b395c5b9" (UID: "ca158e2e-5ec3-406b-a326-ad68b395c5b9"). InnerVolumeSpecName "kube-api-access-x7gzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:33:54.777543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.777524    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca158e2e-5ec3-406b-a326-ad68b395c5b9" (UID: "ca158e2e-5ec3-406b-a326-ad68b395c5b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:33:54.777606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.777579    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca158e2e-5ec3-406b-a326-ad68b395c5b9" (UID: "ca158e2e-5ec3-406b-a326-ad68b395c5b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:33:54.875677 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.875637    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-data\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:54.875677 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.875668    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-podinfo\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:54.875677 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.875677    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-envoy\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:54.875677 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.875685    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-credential-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:54.875956 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.875693    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca158e2e-5ec3-406b-a326-ad68b395c5b9-workload-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:54.875956 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.875702    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7gzv\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-kube-api-access-x7gzv\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:54.875956 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.875710    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istiod-ca-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:54.875956 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:54.875718    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca158e2e-5ec3-406b-a326-ad68b395c5b9-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:33:55.294757 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.294725    2582 generic.go:356] "Generic (PLEG): container finished" podID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerID="7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2" exitCode=0
Jul 28 20:33:55.295024 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.294769    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"
Jul 28 20:33:55.295024 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.294808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" event={"ID":"ca158e2e-5ec3-406b-a326-ad68b395c5b9","Type":"ContainerDied","Data":"7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2"}
Jul 28 20:33:55.295024 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.294865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc" event={"ID":"ca158e2e-5ec3-406b-a326-ad68b395c5b9","Type":"ContainerDied","Data":"88733fb56924ed9b166730917d443105c90fee0810c7dbbe042a6c4fbb678265"}
Jul 28 20:33:55.295024 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.294889    2582 scope.go:117] "RemoveContainer" containerID="7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2"
Jul 28 20:33:55.304357 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.304339    2582 scope.go:117] "RemoveContainer" containerID="f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7"
Jul 28 20:33:55.312208 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.312191    2582 scope.go:117] "RemoveContainer" containerID="7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2"
Jul 28 20:33:55.312486 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:33:55.312470    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2\": container with ID starting with 7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2 not found: ID does not exist" containerID="7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2"
Jul 28 20:33:55.312541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.312494    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2"} err="failed to get container status \"7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2\": rpc error: code = NotFound desc = could not find container \"7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2\": container with ID starting with 7a496863152d07921d7f4ee97bde82233c63ec2545eb82d99cf7ba3c933fd7f2 not found: ID does not exist"
Jul 28 20:33:55.312541 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.312511    2582 scope.go:117] "RemoveContainer" containerID="f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7"
Jul 28 20:33:55.312771 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:33:55.312751    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7\": container with ID starting with f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7 not found: ID does not exist" containerID="f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7"
Jul 28 20:33:55.312823 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.312777    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7"} err="failed to get container status \"f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7\": rpc error: code = NotFound desc = could not find container \"f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7\": container with ID starting with f56f19c6ff15e9eb6d5ddc81e5fa48e146531b76d3fb36cb84b3ccd291adc3a7 not found: ID does not exist"
Jul 28 20:33:55.317027 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.317006    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"]
Jul 28 20:33:55.320809 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.320788    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e9e0fd3a-openshift-default-646bd5d676-km5tc"]
Jul 28 20:33:55.845875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:55.845842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:56.081552 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:56.081511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:56.182745 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:56.182654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:56.287357 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:56.287322    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:56.583322 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:33:56.583274    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" path="/var/lib/kubelet/pods/ca158e2e-5ec3-406b-a326-ad68b395c5b9/volumes"
Jul 28 20:34:02.044171 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:02.044125    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:02.141775 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:02.141732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:08.895443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:08.895407    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:09.200047 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.199958    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"]
Jul 28 20:34:09.200352 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200338    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d64be69-4ca5-4d8d-bba6-fff0c4769db1" containerName="payload-processing"
Jul 28 20:34:09.200352 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200352    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d64be69-4ca5-4d8d-bba6-fff0c4769db1" containerName="payload-processing"
Jul 28 20:34:09.200474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200369    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" containerName="maas-api"
Jul 28 20:34:09.200474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200375    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" containerName="maas-api"
Jul 28 20:34:09.200474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200384    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy"
Jul 28 20:34:09.200474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200390    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy"
Jul 28 20:34:09.200474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200401    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy"
Jul 28 20:34:09.200474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200408    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy"
Jul 28 20:34:09.200474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200466    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d64be69-4ca5-4d8d-bba6-fff0c4769db1" containerName="payload-processing"
Jul 28 20:34:09.200474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200475    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cafe7fec-c0b9-48d0-8c3e-cb492f6f1d99" containerName="maas-api"
Jul 28 20:34:09.200854 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200481    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy"
Jul 28 20:34:09.200854 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.200487    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca158e2e-5ec3-406b-a326-ad68b395c5b9" containerName="istio-proxy"
Jul 28 20:34:09.204806 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.204784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.208809 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.208777    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-404fe153-openshift-default-dockercfg-6ljxg\""
Jul 28 20:34:09.239948 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.239909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:09.268152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.268114    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"]
Jul 28 20:34:09.305938 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.305898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-token\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.306114 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.305942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-socket\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.306114 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.305968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-podinfo\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.306114 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.306063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-data\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.306230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.306119    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-envoy\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.306230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.306170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc351edf-37e5-4d08-a998-17c57084c9ca-istiod-ca-cert\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.306230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.306208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-credential-socket\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.306361 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.306281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-certs\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.306361 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.306339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb57k\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-kube-api-access-cb57k\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.406889 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.406844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-socket\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.406889 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.406892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-podinfo\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407166 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.406919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-data\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407166 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.406951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-envoy\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407166 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.406983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc351edf-37e5-4d08-a998-17c57084c9ca-istiod-ca-cert\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407166 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.407004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-credential-socket\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407166 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.407055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-certs\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407166 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.407140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb57k\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-kube-api-access-cb57k\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407522 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.407224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-token\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407522 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.407348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-data\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407522 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.407477    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-certs\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407722 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.407668    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-socket\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407776 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.407714    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-credential-socket\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.407814 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.407771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc351edf-37e5-4d08-a998-17c57084c9ca-istiod-ca-cert\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.409869 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.409847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-envoy\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.409997 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.409979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-podinfo\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.414012 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.413978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:09.416111 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.416091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-token\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.416231 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.416138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb57k\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-kube-api-access-cb57k\") pod \"e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") " pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.515946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.515912    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:09.656350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.656322    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"]
Jul 28 20:34:09.657742 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:34:09.657717    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc351edf_37e5_4d08_a998_17c57084c9ca.slice/crio-c00e1b46ed3db35ea344a7b1086009abd1502c9887b1e98082968a0b6b21e25e WatchSource:0}: Error finding container c00e1b46ed3db35ea344a7b1086009abd1502c9887b1e98082968a0b6b21e25e: Status 404 returned error can't find the container with id c00e1b46ed3db35ea344a7b1086009abd1502c9887b1e98082968a0b6b21e25e
Jul 28 20:34:09.660153 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.660119    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:34:09.660243 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.660181    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:34:09.660243 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:09.660217    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:34:10.370671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:10.370633    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" event={"ID":"fc351edf-37e5-4d08-a998-17c57084c9ca","Type":"ContainerStarted","Data":"ab64bbaa69f706128562fe338d31160e741b9285f3ec30666f1c5b3be3bd387e"}
Jul 28 20:34:10.371199 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:10.370685    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" event={"ID":"fc351edf-37e5-4d08-a998-17c57084c9ca","Type":"ContainerStarted","Data":"c00e1b46ed3db35ea344a7b1086009abd1502c9887b1e98082968a0b6b21e25e"}
Jul 28 20:34:10.393863 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:10.393812    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" podStartSLOduration=1.39379623 podStartE2EDuration="1.39379623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:34:10.391931251 +0000 UTC m=+2100.246242316" watchObservedRunningTime="2026-07-28 20:34:10.39379623 +0000 UTC m=+2100.248107295"
Jul 28 20:34:10.516609 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:10.516570    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:10.521140 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:10.521115    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:11.375936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:11.375905    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:11.376981 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:11.376962    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:12.388394 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.388360    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:12.502510 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.502475    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"]
Jul 28 20:34:12.507260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.507242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:12.510383 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.510357    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-404fe153-serving-cert\""
Jul 28 20:34:12.526263 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.526234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"]
Jul 28 20:34:12.543973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.543939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:12.639396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.639272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23939582-bb3f-48bf-b422-17e58fcffd39-openshift-service-ca\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:12.639585 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.639394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbbc\" (UniqueName: \"kubernetes.io/projected/23939582-bb3f-48bf-b422-17e58fcffd39-kube-api-access-njbbc\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:12.639657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.639633    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23939582-bb3f-48bf-b422-17e58fcffd39-maas-api-tls\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:12.644782 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.644738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:12.740906 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.740872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23939582-bb3f-48bf-b422-17e58fcffd39-maas-api-tls\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:12.741133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.740924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23939582-bb3f-48bf-b422-17e58fcffd39-openshift-service-ca\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:12.741133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.740982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbbc\" (UniqueName: \"kubernetes.io/projected/23939582-bb3f-48bf-b422-17e58fcffd39-kube-api-access-njbbc\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:12.741133 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:34:12.741036    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-serving-cert: secret "maas-api-e2e-mt-404fe153-serving-cert" not found
Jul 28 20:34:12.741133 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:34:12.741096    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23939582-bb3f-48bf-b422-17e58fcffd39-maas-api-tls podName:23939582-bb3f-48bf-b422-17e58fcffd39 nodeName:}" failed. No retries permitted until 2026-07-28 20:34:13.241080307 +0000 UTC m=+2103.095391351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23939582-bb3f-48bf-b422-17e58fcffd39-maas-api-tls") pod "maas-api-e2e-mt-404fe153-7999c8d856-xfj8j" (UID: "23939582-bb3f-48bf-b422-17e58fcffd39") : secret "maas-api-e2e-mt-404fe153-serving-cert" not found
Jul 28 20:34:12.741810 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.741787    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23939582-bb3f-48bf-b422-17e58fcffd39-openshift-service-ca\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:12.755869 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:12.755840    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbbc\" (UniqueName: \"kubernetes.io/projected/23939582-bb3f-48bf-b422-17e58fcffd39-kube-api-access-njbbc\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:13.041362 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:13.041328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:13.245352 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:13.245285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23939582-bb3f-48bf-b422-17e58fcffd39-maas-api-tls\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:13.247700 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:13.247675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23939582-bb3f-48bf-b422-17e58fcffd39-maas-api-tls\") pod \"maas-api-e2e-mt-404fe153-7999c8d856-xfj8j\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:13.408173 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:13.408077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:13.419919 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:13.419882    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:13.504020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:13.503990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:13.579840 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:13.579814    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"]
Jul 28 20:34:13.582450 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:34:13.582409    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23939582_bb3f_48bf_b422_17e58fcffd39.slice/crio-1882614dfa2cce7722cae04c8b6c760e0bef7c29225aad274189ebb5d71ac968 WatchSource:0}: Error finding container 1882614dfa2cce7722cae04c8b6c760e0bef7c29225aad274189ebb5d71ac968: Status 404 returned error can't find the container with id 1882614dfa2cce7722cae04c8b6c760e0bef7c29225aad274189ebb5d71ac968
Jul 28 20:34:13.584121 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:13.584099    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:34:14.391245 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:14.391157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j" event={"ID":"23939582-bb3f-48bf-b422-17e58fcffd39","Type":"ContainerStarted","Data":"52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61"}
Jul 28 20:34:14.391245 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:14.391208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j" event={"ID":"23939582-bb3f-48bf-b422-17e58fcffd39","Type":"ContainerStarted","Data":"1882614dfa2cce7722cae04c8b6c760e0bef7c29225aad274189ebb5d71ac968"}
Jul 28 20:34:14.391245 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:14.391240    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:14.431171 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:14.431118    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j" podStartSLOduration=1.91453976 podStartE2EDuration="2.431100477s" totalImagesPullingTime="516.560717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:34:14.429355723 +0000 UTC m=+2104.283666789" watchObservedRunningTime="2026-07-28 20:34:14.431100477 +0000 UTC m=+2104.285411591"
Jul 28 20:34:20.402453 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:20.402418    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:34:22.556396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:22.556355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:22.657621 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:22.657587    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:26.377446 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:26.377412    2582 patch_prober.go:28] interesting pod/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 20:34:26.378046 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:26.377482    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 20:34:26.454882 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:26.454849    2582 generic.go:356] "Generic (PLEG): container finished" podID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerID="ab64bbaa69f706128562fe338d31160e741b9285f3ec30666f1c5b3be3bd387e" exitCode=0
Jul 28 20:34:26.455046 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:26.454920    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" event={"ID":"fc351edf-37e5-4d08-a998-17c57084c9ca","Type":"ContainerDied","Data":"ab64bbaa69f706128562fe338d31160e741b9285f3ec30666f1c5b3be3bd387e"}
Jul 28 20:34:26.455284 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:26.455271    2582 scope.go:117] "RemoveContainer" containerID="ab64bbaa69f706128562fe338d31160e741b9285f3ec30666f1c5b3be3bd387e"
Jul 28 20:34:26.456350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:26.456321    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:34:26.456458 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:26.456387    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:34:26.456458 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:26.456417    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:34:27.461800 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:27.461766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" event={"ID":"fc351edf-37e5-4d08-a998-17c57084c9ca","Type":"ContainerStarted","Data":"512079386aebfb52c9e8bc1edcfc49a95eddf68f4be744b49ad7b5a3d611f3ce"}
Jul 28 20:34:27.516437 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:27.516407    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:27.516639 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:27.516617    2582 patch_prober.go:28] interesting pod/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 20:34:27.516688 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:27.516674    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 20:34:28.517268 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:28.517229    2582 patch_prober.go:28] interesting pod/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 20:34:28.517748 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:28.517303    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 20:34:29.516766 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:29.516730    2582 patch_prober.go:28] interesting pod/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 20:34:29.516938 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:29.516809    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 20:34:30.517038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:30.516996    2582 patch_prober.go:28] interesting pod/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 20:34:30.517482 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:30.517071    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 20:34:31.516903 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:31.516868    2582 patch_prober.go:28] interesting pod/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 20:34:31.517078 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:31.516924    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 20:34:32.520868 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:32.520838    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:32.521253 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:32.521209    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:32.521981 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:32.521963    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:34:37.352627 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:37.352590    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:57.629027 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:34:57.628977    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:27.845833 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:27.845794    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"]
Jul 28 20:35:27.849455 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:27.849435    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"
Jul 28 20:35:27.853144 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:27.853124    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\""
Jul 28 20:35:27.859120 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:27.859094    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"]
Jul 28 20:35:27.930140 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:27.930099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a126f954-89ad-45ca-8048-2beb8e6e2fc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-404fe153-mbkfd\" (UID: \"a126f954-89ad-45ca-8048-2beb8e6e2fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"
Jul 28 20:35:28.030996 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:28.030951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a126f954-89ad-45ca-8048-2beb8e6e2fc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-404fe153-mbkfd\" (UID: \"a126f954-89ad-45ca-8048-2beb8e6e2fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"
Jul 28 20:35:28.031810 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:28.031786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a126f954-89ad-45ca-8048-2beb8e6e2fc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-404fe153-mbkfd\" (UID: \"a126f954-89ad-45ca-8048-2beb8e6e2fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"
Jul 28 20:35:28.161217 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:28.161126    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"
Jul 28 20:35:28.296041 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:28.296013    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"]
Jul 28 20:35:28.298857 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:35:28.298827    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda126f954_89ad_45ca_8048_2beb8e6e2fc9.slice/crio-18ed03bff5fb03b57552592a29d9ba2af73ededacaa4d38da2b7e79a5e3e31ce WatchSource:0}: Error finding container 18ed03bff5fb03b57552592a29d9ba2af73ededacaa4d38da2b7e79a5e3e31ce: Status 404 returned error can't find the container with id 18ed03bff5fb03b57552592a29d9ba2af73ededacaa4d38da2b7e79a5e3e31ce
Jul 28 20:35:28.753739 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:28.753700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd" event={"ID":"a126f954-89ad-45ca-8048-2beb8e6e2fc9","Type":"ContainerStarted","Data":"25fe125e67d761b4078343286c3dd62125524a9e26af5292b75818bf25833263"}
Jul 28 20:35:28.754015 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:28.753758    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd" event={"ID":"a126f954-89ad-45ca-8048-2beb8e6e2fc9","Type":"ContainerStarted","Data":"18ed03bff5fb03b57552592a29d9ba2af73ededacaa4d38da2b7e79a5e3e31ce"}
Jul 28 20:35:28.770228 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:28.770179    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd" podStartSLOduration=1.770164706 podStartE2EDuration="1.770164706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:35:28.769225275 +0000 UTC m=+2178.623536340" watchObservedRunningTime="2026-07-28 20:35:28.770164706 +0000 UTC m=+2178.624475772"
Jul 28 20:35:29.760246 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:29.760213    2582 generic.go:356] "Generic (PLEG): container finished" podID="a126f954-89ad-45ca-8048-2beb8e6e2fc9" containerID="25fe125e67d761b4078343286c3dd62125524a9e26af5292b75818bf25833263" exitCode=0
Jul 28 20:35:29.760662 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:29.760279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd" event={"ID":"a126f954-89ad-45ca-8048-2beb8e6e2fc9","Type":"ContainerDied","Data":"25fe125e67d761b4078343286c3dd62125524a9e26af5292b75818bf25833263"}
Jul 28 20:35:30.899033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:30.899007    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"
Jul 28 20:35:30.955649 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:30.955620    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a126f954-89ad-45ca-8048-2beb8e6e2fc9-maas-api-service-ca\") pod \"a126f954-89ad-45ca-8048-2beb8e6e2fc9\" (UID: \"a126f954-89ad-45ca-8048-2beb8e6e2fc9\") "
Jul 28 20:35:30.955996 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:30.955972    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a126f954-89ad-45ca-8048-2beb8e6e2fc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a126f954-89ad-45ca-8048-2beb8e6e2fc9" (UID: "a126f954-89ad-45ca-8048-2beb8e6e2fc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:35:31.057110 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:31.057021    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a126f954-89ad-45ca-8048-2beb8e6e2fc9-maas-api-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:35:31.770260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:31.770226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd" event={"ID":"a126f954-89ad-45ca-8048-2beb8e6e2fc9","Type":"ContainerDied","Data":"18ed03bff5fb03b57552592a29d9ba2af73ededacaa4d38da2b7e79a5e3e31ce"}
Jul 28 20:35:31.770260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:31.770255    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"
Jul 28 20:35:31.770491 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:31.770274    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18ed03bff5fb03b57552592a29d9ba2af73ededacaa4d38da2b7e79a5e3e31ce"
Jul 28 20:35:43.407171 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.407134    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"]
Jul 28 20:35:43.407702 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.407382    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j" podUID="23939582-bb3f-48bf-b422-17e58fcffd39" containerName="maas-api" containerID="cri-o://52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61" gracePeriod=30
Jul 28 20:35:43.435497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.435469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:43.537621 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.537584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:43.636389 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.636315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:43.647354 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.647318    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"]
Jul 28 20:35:43.650672 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.650644    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-404fe153-mbkfd"]
Jul 28 20:35:43.663078 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.663013    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:35:43.739553 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.739518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:43.770572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.770543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njbbc\" (UniqueName: \"kubernetes.io/projected/23939582-bb3f-48bf-b422-17e58fcffd39-kube-api-access-njbbc\") pod \"23939582-bb3f-48bf-b422-17e58fcffd39\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") "
Jul 28 20:35:43.770767 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.770689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23939582-bb3f-48bf-b422-17e58fcffd39-openshift-service-ca\") pod \"23939582-bb3f-48bf-b422-17e58fcffd39\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") "
Jul 28 20:35:43.770767 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.770723    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23939582-bb3f-48bf-b422-17e58fcffd39-maas-api-tls\") pod \"23939582-bb3f-48bf-b422-17e58fcffd39\" (UID: \"23939582-bb3f-48bf-b422-17e58fcffd39\") "
Jul 28 20:35:43.771065 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.771042    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23939582-bb3f-48bf-b422-17e58fcffd39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23939582-bb3f-48bf-b422-17e58fcffd39" (UID: "23939582-bb3f-48bf-b422-17e58fcffd39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:35:43.772733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.772710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23939582-bb3f-48bf-b422-17e58fcffd39-kube-api-access-njbbc" (OuterVolumeSpecName: "kube-api-access-njbbc") pod "23939582-bb3f-48bf-b422-17e58fcffd39" (UID: "23939582-bb3f-48bf-b422-17e58fcffd39"). InnerVolumeSpecName "kube-api-access-njbbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:35:43.772821 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.772743    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23939582-bb3f-48bf-b422-17e58fcffd39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23939582-bb3f-48bf-b422-17e58fcffd39" (UID: "23939582-bb3f-48bf-b422-17e58fcffd39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:35:43.828742 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.828705    2582 generic.go:356] "Generic (PLEG): container finished" podID="23939582-bb3f-48bf-b422-17e58fcffd39" containerID="52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61" exitCode=0
Jul 28 20:35:43.828941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.828756    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"
Jul 28 20:35:43.828941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.828800    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j" event={"ID":"23939582-bb3f-48bf-b422-17e58fcffd39","Type":"ContainerDied","Data":"52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61"}
Jul 28 20:35:43.828941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.828867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j" event={"ID":"23939582-bb3f-48bf-b422-17e58fcffd39","Type":"ContainerDied","Data":"1882614dfa2cce7722cae04c8b6c760e0bef7c29225aad274189ebb5d71ac968"}
Jul 28 20:35:43.828941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.828902    2582 scope.go:117] "RemoveContainer" containerID="52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61"
Jul 28 20:35:43.838497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.838474    2582 scope.go:117] "RemoveContainer" containerID="52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61"
Jul 28 20:35:43.838777 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:35:43.838758    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61\": container with ID starting with 52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61 not found: ID does not exist" containerID="52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61"
Jul 28 20:35:43.838812 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.838788    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61"} err="failed to get container status \"52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61\": rpc error: code = NotFound desc = could not find container \"52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61\": container with ID starting with 52cec3ce5178502d68a1daf051617cbebc4fda88d2ba40d99ba791489fc78c61 not found: ID does not exist"
Jul 28 20:35:43.853646 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.853610    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"]
Jul 28 20:35:43.856565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.856540    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-404fe153-7999c8d856-xfj8j"]
Jul 28 20:35:43.871764 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.871737    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23939582-bb3f-48bf-b422-17e58fcffd39-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:35:43.871764 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.871768    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23939582-bb3f-48bf-b422-17e58fcffd39-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:35:43.871963 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:43.871782    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njbbc\" (UniqueName: \"kubernetes.io/projected/23939582-bb3f-48bf-b422-17e58fcffd39-kube-api-access-njbbc\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:35:44.583407 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:44.583377    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23939582-bb3f-48bf-b422-17e58fcffd39" path="/var/lib/kubelet/pods/23939582-bb3f-48bf-b422-17e58fcffd39/volumes"
Jul 28 20:35:44.583798 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:44.583699    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a126f954-89ad-45ca-8048-2beb8e6e2fc9" path="/var/lib/kubelet/pods/a126f954-89ad-45ca-8048-2beb8e6e2fc9/volumes"
Jul 28 20:35:47.638472 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:47.638438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:52.665031 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:35:52.664995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:26.090893 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:26.090852    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:27.247304 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:27.247247    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:28.853219 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:28.853182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:29.925409 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:29.925368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:35.626490 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:35.626369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:35.692978 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:35.692938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:35.900913 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:35.900825    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"]
Jul 28 20:36:35.901142 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:35.901111    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy" containerID="cri-o://512079386aebfb52c9e8bc1edcfc49a95eddf68f4be744b49ad7b5a3d611f3ce" gracePeriod=30
Jul 28 20:36:35.937734 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:35.937692    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:36.168570 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:36.168462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:37.278489 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:37.278438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:37.409079 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:37.409047    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:38.207158 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:38.207121    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:38.317204 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:38.317158    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:38.413730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:38.413693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:39.832530 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:39.832493    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:41.100809 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.100448    2582 generic.go:356] "Generic (PLEG): container finished" podID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerID="512079386aebfb52c9e8bc1edcfc49a95eddf68f4be744b49ad7b5a3d611f3ce" exitCode=0
Jul 28 20:36:41.100809 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.100614    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" event={"ID":"fc351edf-37e5-4d08-a998-17c57084c9ca","Type":"ContainerDied","Data":"512079386aebfb52c9e8bc1edcfc49a95eddf68f4be744b49ad7b5a3d611f3ce"}
Jul 28 20:36:41.100809 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.100678    2582 scope.go:117] "RemoveContainer" containerID="ab64bbaa69f706128562fe338d31160e741b9285f3ec30666f1c5b3be3bd387e"
Jul 28 20:36:41.174659 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.174588    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:36:41.287883 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.287853    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb57k\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-kube-api-access-cb57k\") pod \"fc351edf-37e5-4d08-a998-17c57084c9ca\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") "
Jul 28 20:36:41.288049 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.287901    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc351edf-37e5-4d08-a998-17c57084c9ca-istiod-ca-cert\") pod \"fc351edf-37e5-4d08-a998-17c57084c9ca\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") "
Jul 28 20:36:41.288049 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.287967    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-socket\") pod \"fc351edf-37e5-4d08-a998-17c57084c9ca\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") "
Jul 28 20:36:41.288049 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.287995    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-data\") pod \"fc351edf-37e5-4d08-a998-17c57084c9ca\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") "
Jul 28 20:36:41.288049 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-credential-socket\") pod \"fc351edf-37e5-4d08-a998-17c57084c9ca\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") "
Jul 28 20:36:41.288049 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288033    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-certs\") pod \"fc351edf-37e5-4d08-a998-17c57084c9ca\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") "
Jul 28 20:36:41.288333 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288056    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-podinfo\") pod \"fc351edf-37e5-4d08-a998-17c57084c9ca\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") "
Jul 28 20:36:41.288333 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288078    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-envoy\") pod \"fc351edf-37e5-4d08-a998-17c57084c9ca\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") "
Jul 28 20:36:41.288333 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288107    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-token\") pod \"fc351edf-37e5-4d08-a998-17c57084c9ca\" (UID: \"fc351edf-37e5-4d08-a998-17c57084c9ca\") "
Jul 28 20:36:41.288333 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288209    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc351edf-37e5-4d08-a998-17c57084c9ca" (UID: "fc351edf-37e5-4d08-a998-17c57084c9ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:36:41.288532 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288342    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc351edf-37e5-4d08-a998-17c57084c9ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc351edf-37e5-4d08-a998-17c57084c9ca" (UID: "fc351edf-37e5-4d08-a998-17c57084c9ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:36:41.288532 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288351    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc351edf-37e5-4d08-a998-17c57084c9ca" (UID: "fc351edf-37e5-4d08-a998-17c57084c9ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:36:41.288657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288528    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:36:41.288657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288548    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-credential-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:36:41.288657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288544    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc351edf-37e5-4d08-a998-17c57084c9ca" (UID: "fc351edf-37e5-4d08-a998-17c57084c9ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:36:41.288657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.288564    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc351edf-37e5-4d08-a998-17c57084c9ca-istiod-ca-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:36:41.289275 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.289243    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc351edf-37e5-4d08-a998-17c57084c9ca" (UID: "fc351edf-37e5-4d08-a998-17c57084c9ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:36:41.290366 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.290282    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-kube-api-access-cb57k" (OuterVolumeSpecName: "kube-api-access-cb57k") pod "fc351edf-37e5-4d08-a998-17c57084c9ca" (UID: "fc351edf-37e5-4d08-a998-17c57084c9ca"). InnerVolumeSpecName "kube-api-access-cb57k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:36:41.290453 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.290397    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc351edf-37e5-4d08-a998-17c57084c9ca" (UID: "fc351edf-37e5-4d08-a998-17c57084c9ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:36:41.290453 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.290424    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc351edf-37e5-4d08-a998-17c57084c9ca" (UID: "fc351edf-37e5-4d08-a998-17c57084c9ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:36:41.290584 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.290563    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc351edf-37e5-4d08-a998-17c57084c9ca" (UID: "fc351edf-37e5-4d08-a998-17c57084c9ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:36:41.389958 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.389921    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb57k\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-kube-api-access-cb57k\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:36:41.389958 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.389951    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-data\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:36:41.389958 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.389961    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-workload-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:36:41.389958 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.389970    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-podinfo\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:36:41.390257 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.389978    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-envoy\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:36:41.390257 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:41.389987    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc351edf-37e5-4d08-a998-17c57084c9ca-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:36:42.115191 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:42.115164    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"
Jul 28 20:36:42.115712 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:42.115156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68" event={"ID":"fc351edf-37e5-4d08-a998-17c57084c9ca","Type":"ContainerDied","Data":"c00e1b46ed3db35ea344a7b1086009abd1502c9887b1e98082968a0b6b21e25e"}
Jul 28 20:36:42.115712 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:42.115356    2582 scope.go:117] "RemoveContainer" containerID="512079386aebfb52c9e8bc1edcfc49a95eddf68f4be744b49ad7b5a3d611f3ce"
Jul 28 20:36:42.144847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:42.144812    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"]
Jul 28 20:36:42.147594 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:42.147570    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-404fe153-openshift-default-5bcf5f4896-djj68"]
Jul 28 20:36:42.584249 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:42.584207    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" path="/var/lib/kubelet/pods/fc351edf-37e5-4d08-a998-17c57084c9ca/volumes"
Jul 28 20:36:45.551075 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:45.551029    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:45.652778 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:45.652737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:46.288267 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:46.288229    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:49.000573 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:49.000535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:52.263449 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:52.263398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:52.372742 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:52.372704    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:52.472756 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:36:52.472719    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:03.112007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:03.111956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:03.439549 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:03.439430    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:10.835926 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:10.835889    2582 scope.go:117] "RemoveContainer" containerID="9fc0fabc01d37e12649e6418ba2cf728103c3daad50c656852411c8d9fe86522"
Jul 28 20:37:24.438064 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:24.438023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:31.137276 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.137225    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:31.237898 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.237851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:31.520384 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.520341    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:31.854146 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854067    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"]
Jul 28 20:37:31.854475 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854463    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy"
Jul 28 20:37:31.854523 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854477    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy"
Jul 28 20:37:31.854523 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854484    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a126f954-89ad-45ca-8048-2beb8e6e2fc9" containerName="revoke-keys"
Jul 28 20:37:31.854523 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854490    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a126f954-89ad-45ca-8048-2beb8e6e2fc9" containerName="revoke-keys"
Jul 28 20:37:31.854523 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854505    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23939582-bb3f-48bf-b422-17e58fcffd39" containerName="maas-api"
Jul 28 20:37:31.854523 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854510    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="23939582-bb3f-48bf-b422-17e58fcffd39" containerName="maas-api"
Jul 28 20:37:31.854523 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854519    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy"
Jul 28 20:37:31.854523 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854524    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy"
Jul 28 20:37:31.854741 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854580    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a126f954-89ad-45ca-8048-2beb8e6e2fc9" containerName="revoke-keys"
Jul 28 20:37:31.854741 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854588    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="23939582-bb3f-48bf-b422-17e58fcffd39" containerName="maas-api"
Jul 28 20:37:31.854741 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854596    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy"
Jul 28 20:37:31.854741 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.854602    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc351edf-37e5-4d08-a998-17c57084c9ca" containerName="istio-proxy"
Jul 28 20:37:31.857091 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.857064    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:31.859986 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.859964    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7ff6b3-openshift-default-dockercfg-4sc4b\""
Jul 28 20:37:31.870162 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.870138    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"]
Jul 28 20:37:31.876486 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.876460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:31.932573 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.932535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-envoy\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:31.932780 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.932628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gffmz\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-kube-api-access-gffmz\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:31.932780 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.932678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-certs\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:31.932780 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.932740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-podinfo\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:31.932780 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.932768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20c43c4-dfea-499d-9d04-28506a5bf109-istiod-ca-cert\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:31.932978 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.932794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-credential-socket\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:31.932978 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.932826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-socket\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:31.932978 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.932932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-data\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:31.933090 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:31.932984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-token\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034085 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-data\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034279 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-token\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034279 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-envoy\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034279 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gffmz\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-kube-api-access-gffmz\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034279 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-certs\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034279 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-podinfo\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034279 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20c43c4-dfea-499d-9d04-28506a5bf109-istiod-ca-cert\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034634 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-credential-socket\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034697 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-certs\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034753 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034721    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-socket\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.034933 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034911    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-credential-socket\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.035022 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.034937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-socket\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.035078 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.035026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20c43c4-dfea-499d-9d04-28506a5bf109-istiod-ca-cert\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.035118 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.035075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-data\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.036928 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.036893    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-envoy\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.037139 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.037118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-podinfo\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.042069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.042023    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-token\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.046191 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.046168    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gffmz\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-kube-api-access-gffmz\") pod \"e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") " pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.093033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.092998    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:32.168872 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.168777    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:32.314197 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.314168    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"]
Jul 28 20:37:32.316348 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:37:32.316321    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20c43c4_dfea_499d_9d04_28506a5bf109.slice/crio-18d0d4ccab31711d7e472d3107448679d4005ce51e112c9d93ab76c860efa595 WatchSource:0}: Error finding container 18d0d4ccab31711d7e472d3107448679d4005ce51e112c9d93ab76c860efa595: Status 404 returned error can't find the container with id 18d0d4ccab31711d7e472d3107448679d4005ce51e112c9d93ab76c860efa595
Jul 28 20:37:32.319396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.319285    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:37:32.319526 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.319465    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:37:32.319598 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.319555    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:37:32.359611 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:32.359583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" event={"ID":"e20c43c4-dfea-499d-9d04-28506a5bf109","Type":"ContainerStarted","Data":"18d0d4ccab31711d7e472d3107448679d4005ce51e112c9d93ab76c860efa595"}
Jul 28 20:37:33.366322 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:33.366267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" event={"ID":"e20c43c4-dfea-499d-9d04-28506a5bf109","Type":"ContainerStarted","Data":"dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c"}
Jul 28 20:37:33.392991 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:33.392927    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" podStartSLOduration=2.392909244 podStartE2EDuration="2.392909244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:37:33.391323575 +0000 UTC m=+2303.245634655" watchObservedRunningTime="2026-07-28 20:37:33.392909244 +0000 UTC m=+2303.247220310"
Jul 28 20:37:34.169698 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:34.169660    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:34.174889 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:34.174861    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:34.372004 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:34.371974    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:34.373146 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:34.373124    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:35.474277 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.474209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:35.574715 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.574679    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:35.749842 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.749811    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"]
Jul 28 20:37:35.759021 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.758993    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:35.762606 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.762573    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7ff6b3-serving-cert\""
Jul 28 20:37:35.773316 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.773272    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"]
Jul 28 20:37:35.811555 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.811521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:35.871597 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.871559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6893a4e-1f1d-4931-a76e-e380be4fd804-maas-api-tls\") pod \"maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:35.871774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.871622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6893a4e-1f1d-4931-a76e-e380be4fd804-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:35.871774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.871646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8xzb\" (UniqueName: \"kubernetes.io/projected/d6893a4e-1f1d-4931-a76e-e380be4fd804-kube-api-access-n8xzb\") pod \"maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:35.972779 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.972739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6893a4e-1f1d-4931-a76e-e380be4fd804-maas-api-tls\") pod \"maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:35.972998 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.972810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6893a4e-1f1d-4931-a76e-e380be4fd804-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:35.972998 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.972835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xzb\" (UniqueName: \"kubernetes.io/projected/d6893a4e-1f1d-4931-a76e-e380be4fd804-kube-api-access-n8xzb\") pod \"maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:35.973610 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.973580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6893a4e-1f1d-4931-a76e-e380be4fd804-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:35.973820 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.973751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:35.975353 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.975337    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6893a4e-1f1d-4931-a76e-e380be4fd804-maas-api-tls\") pod \"maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:35.988711 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:35.988684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xzb\" (UniqueName: \"kubernetes.io/projected/d6893a4e-1f1d-4931-a76e-e380be4fd804-kube-api-access-n8xzb\") pod \"maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:36.071437 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:36.071339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:36.242793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:36.242768    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"]
Jul 28 20:37:36.245140 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:37:36.245109    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6893a4e_1f1d_4931_a76e_e380be4fd804.slice/crio-d24a0dd8f5c9472ac2b5b76ac82232cf31c953639f08cc5c5a89decca3a0efe7 WatchSource:0}: Error finding container d24a0dd8f5c9472ac2b5b76ac82232cf31c953639f08cc5c5a89decca3a0efe7: Status 404 returned error can't find the container with id d24a0dd8f5c9472ac2b5b76ac82232cf31c953639f08cc5c5a89decca3a0efe7
Jul 28 20:37:36.382525 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:36.382432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g" event={"ID":"d6893a4e-1f1d-4931-a76e-e380be4fd804","Type":"ContainerStarted","Data":"d24a0dd8f5c9472ac2b5b76ac82232cf31c953639f08cc5c5a89decca3a0efe7"}
Jul 28 20:37:36.707050 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:36.706960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:36.805798 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:36.805754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:38.396100 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:38.396064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g" event={"ID":"d6893a4e-1f1d-4931-a76e-e380be4fd804","Type":"ContainerStarted","Data":"f1da379a8441593106d52ad7c326e4afb119648015f9350b9f30a27ff5884457"}
Jul 28 20:37:38.396531 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:38.396127    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:38.430909 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:38.430806    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g" podStartSLOduration=1.519245448 podStartE2EDuration="3.430790423s" totalImagesPullingTime="1.911544975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:37:38.428835405 +0000 UTC m=+2308.283146472" watchObservedRunningTime="2026-07-28 20:37:38.430790423 +0000 UTC m=+2308.285101489"
Jul 28 20:37:44.407307 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:44.407264    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:37:49.373319 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:49.373227    2582 patch_prober.go:28] interesting pod/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 28 20:37:49.373697 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:49.373286    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 28 20:37:49.455057 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:49.455026    2582 generic.go:356] "Generic (PLEG): container finished" podID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerID="dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c" exitCode=0
Jul 28 20:37:49.455227 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:49.455100    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" event={"ID":"e20c43c4-dfea-499d-9d04-28506a5bf109","Type":"ContainerDied","Data":"dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c"}
Jul 28 20:37:49.455572 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:49.455554    2582 scope.go:117] "RemoveContainer" containerID="dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c"
Jul 28 20:37:49.456678 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:49.456652    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:37:49.456737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:49.456718    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:37:49.456776 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:49.456756    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:37:50.462739 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:50.462704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" event={"ID":"e20c43c4-dfea-499d-9d04-28506a5bf109","Type":"ContainerStarted","Data":"6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46"}
Jul 28 20:37:51.169330 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:51.169271    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:51.169664 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:51.169639    2582 patch_prober.go:28] interesting pod/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 28 20:37:51.169773 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:51.169694    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 28 20:37:52.169675 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:52.169637    2582 patch_prober.go:28] interesting pod/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 28 20:37:52.170091 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:52.169712    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 28 20:37:53.169578 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:53.169545    2582 patch_prober.go:28] interesting pod/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 28 20:37:53.169758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:53.169598    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 28 20:37:54.169847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:54.169811    2582 patch_prober.go:28] interesting pod/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 28 20:37:54.170271 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:54.169868    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 28 20:37:55.173119 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:55.173085    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:55.173588 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:55.173473    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:37:55.174228 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:37:55.174212    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:38:00.382789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:00.382744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:00.769301 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:00.769265    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:01.055024 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:01.054925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:03.135779 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:03.135695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:04.508608 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.508570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:04.605781 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.605743    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:04.614250 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.614217    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"]
Jul 28 20:38:04.618514 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.618492    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:04.622043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.622019    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d38c0f-serving-cert\""
Jul 28 20:38:04.622371 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.622343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569312a2-1ab6-4fc5-8007-f921b593adfa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:04.622502 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.622405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569312a2-1ab6-4fc5-8007-f921b593adfa-maas-api-tls\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:04.622564 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.622542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzd2\" (UniqueName: \"kubernetes.io/projected/569312a2-1ab6-4fc5-8007-f921b593adfa-kube-api-access-ftzd2\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:04.631975 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.631946    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"]
Jul 28 20:38:04.723312 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.723268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzd2\" (UniqueName: \"kubernetes.io/projected/569312a2-1ab6-4fc5-8007-f921b593adfa-kube-api-access-ftzd2\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:04.723519 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.723378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569312a2-1ab6-4fc5-8007-f921b593adfa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:04.723519 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.723428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569312a2-1ab6-4fc5-8007-f921b593adfa-maas-api-tls\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:04.723639 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:38:04.723559    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-serving-cert: secret "maas-api-e2e-inf-b-d38c0f-serving-cert" not found
Jul 28 20:38:04.723639 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:38:04.723620    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569312a2-1ab6-4fc5-8007-f921b593adfa-maas-api-tls podName:569312a2-1ab6-4fc5-8007-f921b593adfa nodeName:}" failed. No retries permitted until 2026-07-28 20:38:05.223600291 +0000 UTC m=+2335.077911350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/569312a2-1ab6-4fc5-8007-f921b593adfa-maas-api-tls") pod "maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g" (UID: "569312a2-1ab6-4fc5-8007-f921b593adfa") : secret "maas-api-e2e-inf-b-d38c0f-serving-cert" not found
Jul 28 20:38:04.725371 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.724752    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569312a2-1ab6-4fc5-8007-f921b593adfa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:04.726793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.726763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:04.734939 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.734908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzd2\" (UniqueName: \"kubernetes.io/projected/569312a2-1ab6-4fc5-8007-f921b593adfa-kube-api-access-ftzd2\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:04.900115 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:04.900027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:05.229122 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:05.229020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569312a2-1ab6-4fc5-8007-f921b593adfa-maas-api-tls\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:05.231641 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:05.231609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569312a2-1ab6-4fc5-8007-f921b593adfa-maas-api-tls\") pod \"maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:05.305750 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:05.305716    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:05.412285 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:05.412246    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:05.530051 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:05.530013    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:05.683440 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:05.683360    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"]
Jul 28 20:38:05.685899 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:38:05.685865    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569312a2_1ab6_4fc5_8007_f921b593adfa.slice/crio-2d2f3d6bf67b5fe2ceb27af37a530821f95d18ea356d8f7b689a68b8be7235b3 WatchSource:0}: Error finding container 2d2f3d6bf67b5fe2ceb27af37a530821f95d18ea356d8f7b689a68b8be7235b3: Status 404 returned error can't find the container with id 2d2f3d6bf67b5fe2ceb27af37a530821f95d18ea356d8f7b689a68b8be7235b3
Jul 28 20:38:06.549278 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:06.549190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g" event={"ID":"569312a2-1ab6-4fc5-8007-f921b593adfa","Type":"ContainerStarted","Data":"a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7"}
Jul 28 20:38:06.549278 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:06.549248    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g" event={"ID":"569312a2-1ab6-4fc5-8007-f921b593adfa","Type":"ContainerStarted","Data":"2d2f3d6bf67b5fe2ceb27af37a530821f95d18ea356d8f7b689a68b8be7235b3"}
Jul 28 20:38:06.549757 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:06.549338    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:06.588729 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:06.588668    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g" podStartSLOduration=1.9851398 podStartE2EDuration="2.588648841s" totalImagesPullingTime="603.509041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:38:06.585278442 +0000 UTC m=+2336.439589540" watchObservedRunningTime="2026-07-28 20:38:06.588648841 +0000 UTC m=+2336.442959908"
Jul 28 20:38:12.559558 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:12.559526    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:38:29.464615 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:29.464574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:40.589809 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.589770    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"]
Jul 28 20:38:40.594070 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.594048    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.598866 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.598838    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7ff6b3\"/\"test-model-7ff6b3-kserve-self-signed-certs\""
Jul 28 20:38:40.598866 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.598856    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7ff6b3\"/\"kube-root-ca.crt\""
Jul 28 20:38:40.599062 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.598918    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7ff6b3\"/\"default-dockercfg-kgzqm\""
Jul 28 20:38:40.599062 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.598918    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7ff6b3\"/\"openshift-service-ca.crt\""
Jul 28 20:38:40.605994 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.605969    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"]
Jul 28 20:38:40.761969 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.761937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpljj\" (UniqueName: \"kubernetes.io/projected/093204b3-1e91-4f1e-a16e-1ed3553be129-kube-api-access-bpljj\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.762153 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.761992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-home\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.762153 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.762069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093204b3-1e91-4f1e-a16e-1ed3553be129-tls-certs\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.762153 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.762105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-model-cache\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.762153 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.762136    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-tmp-dir\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.762400 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.762223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-dshm\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.807739 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.807701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:40.863045 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.862958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpljj\" (UniqueName: \"kubernetes.io/projected/093204b3-1e91-4f1e-a16e-1ed3553be129-kube-api-access-bpljj\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.863045 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.863001    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-home\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.863045 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.863033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093204b3-1e91-4f1e-a16e-1ed3553be129-tls-certs\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.863305 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.863063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-model-cache\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.863305 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.863087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-tmp-dir\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.863305 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.863142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-dshm\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.863480 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.863462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-home\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.863539 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.863491    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-model-cache\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.863594 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.863552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-tmp-dir\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.865355 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.865328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-dshm\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.865548 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.865531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093204b3-1e91-4f1e-a16e-1ed3553be129-tls-certs\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.878192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.878162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpljj\" (UniqueName: \"kubernetes.io/projected/093204b3-1e91-4f1e-a16e-1ed3553be129-kube-api-access-bpljj\") pod \"test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") " pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:40.903358 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:40.903319    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:41.043056 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:41.043023    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"]
Jul 28 20:38:41.045171 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:38:41.045146    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod093204b3_1e91_4f1e_a16e_1ed3553be129.slice/crio-56ba192ac83cb76041b726689c6ed46e7957d70863be73b8f70284c3c7073528 WatchSource:0}: Error finding container 56ba192ac83cb76041b726689c6ed46e7957d70863be73b8f70284c3c7073528: Status 404 returned error can't find the container with id 56ba192ac83cb76041b726689c6ed46e7957d70863be73b8f70284c3c7073528
Jul 28 20:38:41.724356 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:41.724311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg" event={"ID":"093204b3-1e91-4f1e-a16e-1ed3553be129","Type":"ContainerStarted","Data":"49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f"}
Jul 28 20:38:41.724755 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:41.724366    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg" event={"ID":"093204b3-1e91-4f1e-a16e-1ed3553be129","Type":"ContainerStarted","Data":"56ba192ac83cb76041b726689c6ed46e7957d70863be73b8f70284c3c7073528"}
Jul 28 20:38:41.755741 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:41.755686    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg" podStartSLOduration=1.755669736 podStartE2EDuration="1.755669736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:38:41.754468769 +0000 UTC m=+2371.608779835" watchObservedRunningTime="2026-07-28 20:38:41.755669736 +0000 UTC m=+2371.609980801"
Jul 28 20:38:47.216928 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:47.216886    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:50.903483 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:50.903448    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:50.903880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:50.903502    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:50.916397 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:50.916373    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:38:51.791443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:38:51.791417    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:39:48.206896 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:39:48.206804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:01.199384 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:01.199345    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:11.993364 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:11.993326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:40.515622 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:40.515581    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"]
Jul 28 20:40:40.553924 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:40.553892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:41.470544 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:41.470503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:41.553191 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:41.553107    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"]
Jul 28 20:40:41.556890 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:41.556872    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"
Jul 28 20:40:41.559889 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:41.559868    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\""
Jul 28 20:40:41.566912 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:41.566889    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"]
Jul 28 20:40:41.646180 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:41.646147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f69e61-04aa-4326-b50a-03fa13787342-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9\" (UID: \"b7f69e61-04aa-4326-b50a-03fa13787342\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"
Jul 28 20:40:41.747031 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:41.746997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f69e61-04aa-4326-b50a-03fa13787342-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9\" (UID: \"b7f69e61-04aa-4326-b50a-03fa13787342\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"
Jul 28 20:40:41.747663 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:41.747643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f69e61-04aa-4326-b50a-03fa13787342-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9\" (UID: \"b7f69e61-04aa-4326-b50a-03fa13787342\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"
Jul 28 20:40:41.867938 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:41.867905    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"
Jul 28 20:40:42.005188 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:42.005157    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"]
Jul 28 20:40:42.007111 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:40:42.007084    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f69e61_04aa_4326_b50a_03fa13787342.slice/crio-33f98f3a96f1371e0520ff105f02fd7bb9d8efd920425273af2ae7b1c2e046b9 WatchSource:0}: Error finding container 33f98f3a96f1371e0520ff105f02fd7bb9d8efd920425273af2ae7b1c2e046b9: Status 404 returned error can't find the container with id 33f98f3a96f1371e0520ff105f02fd7bb9d8efd920425273af2ae7b1c2e046b9
Jul 28 20:40:42.326146 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:42.326116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9" event={"ID":"b7f69e61-04aa-4326-b50a-03fa13787342","Type":"ContainerStarted","Data":"33f98f3a96f1371e0520ff105f02fd7bb9d8efd920425273af2ae7b1c2e046b9"}
Jul 28 20:40:43.332881 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:43.332846    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7f69e61-04aa-4326-b50a-03fa13787342" containerID="7430a6ea9ff318f59cbfe5ee3a1f8b7436c2c821d27c9aebe6dc41644b5cfe85" exitCode=0
Jul 28 20:40:43.333370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:43.332939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9" event={"ID":"b7f69e61-04aa-4326-b50a-03fa13787342","Type":"ContainerDied","Data":"7430a6ea9ff318f59cbfe5ee3a1f8b7436c2c821d27c9aebe6dc41644b5cfe85"}
Jul 28 20:40:44.475954 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:44.475922    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"
Jul 28 20:40:44.571091 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:44.571050    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f69e61-04aa-4326-b50a-03fa13787342-maas-api-service-ca\") pod \"b7f69e61-04aa-4326-b50a-03fa13787342\" (UID: \"b7f69e61-04aa-4326-b50a-03fa13787342\") "
Jul 28 20:40:44.571422 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:44.571400    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f69e61-04aa-4326-b50a-03fa13787342-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7f69e61-04aa-4326-b50a-03fa13787342" (UID: "b7f69e61-04aa-4326-b50a-03fa13787342"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:40:44.672518 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:44.672481    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f69e61-04aa-4326-b50a-03fa13787342-maas-api-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:45.344445 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:45.344415    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"
Jul 28 20:40:45.344643 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:45.344417    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9" event={"ID":"b7f69e61-04aa-4326-b50a-03fa13787342","Type":"ContainerDied","Data":"33f98f3a96f1371e0520ff105f02fd7bb9d8efd920425273af2ae7b1c2e046b9"}
Jul 28 20:40:45.344643 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:45.344538    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33f98f3a96f1371e0520ff105f02fd7bb9d8efd920425273af2ae7b1c2e046b9"
Jul 28 20:40:52.129098 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.129063    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"]
Jul 28 20:40:52.129865 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.129335    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g" podUID="d6893a4e-1f1d-4931-a76e-e380be4fd804" containerName="maas-api" containerID="cri-o://f1da379a8441593106d52ad7c326e4afb119648015f9350b9f30a27ff5884457" gracePeriod=30
Jul 28 20:40:52.176264 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.176228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:52.287128 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.287091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:52.380710 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.380639    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6893a4e-1f1d-4931-a76e-e380be4fd804" containerID="f1da379a8441593106d52ad7c326e4afb119648015f9350b9f30a27ff5884457" exitCode=0
Jul 28 20:40:52.380828 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.380717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g" event={"ID":"d6893a4e-1f1d-4931-a76e-e380be4fd804","Type":"ContainerDied","Data":"f1da379a8441593106d52ad7c326e4afb119648015f9350b9f30a27ff5884457"}
Jul 28 20:40:52.380828 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.380797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g" event={"ID":"d6893a4e-1f1d-4931-a76e-e380be4fd804","Type":"ContainerDied","Data":"d24a0dd8f5c9472ac2b5b76ac82232cf31c953639f08cc5c5a89decca3a0efe7"}
Jul 28 20:40:52.380828 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.380814    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d24a0dd8f5c9472ac2b5b76ac82232cf31c953639f08cc5c5a89decca3a0efe7"
Jul 28 20:40:52.390093 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.390064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:52.395701 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.395678    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:40:52.409620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.409583    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"]
Jul 28 20:40:52.411567 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.411536    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ff6b3-nl4n9"]
Jul 28 20:40:52.441977 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.441939    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6893a4e-1f1d-4931-a76e-e380be4fd804-openshift-service-ca\") pod \"d6893a4e-1f1d-4931-a76e-e380be4fd804\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") "
Jul 28 20:40:52.442161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.441996    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8xzb\" (UniqueName: \"kubernetes.io/projected/d6893a4e-1f1d-4931-a76e-e380be4fd804-kube-api-access-n8xzb\") pod \"d6893a4e-1f1d-4931-a76e-e380be4fd804\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") "
Jul 28 20:40:52.442161 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.442078    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6893a4e-1f1d-4931-a76e-e380be4fd804-maas-api-tls\") pod \"d6893a4e-1f1d-4931-a76e-e380be4fd804\" (UID: \"d6893a4e-1f1d-4931-a76e-e380be4fd804\") "
Jul 28 20:40:52.442360 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.442334    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6893a4e-1f1d-4931-a76e-e380be4fd804-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6893a4e-1f1d-4931-a76e-e380be4fd804" (UID: "d6893a4e-1f1d-4931-a76e-e380be4fd804"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:40:52.444402 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.444342    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6893a4e-1f1d-4931-a76e-e380be4fd804-kube-api-access-n8xzb" (OuterVolumeSpecName: "kube-api-access-n8xzb") pod "d6893a4e-1f1d-4931-a76e-e380be4fd804" (UID: "d6893a4e-1f1d-4931-a76e-e380be4fd804"). InnerVolumeSpecName "kube-api-access-n8xzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:40:52.444402 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.444362    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6893a4e-1f1d-4931-a76e-e380be4fd804-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6893a4e-1f1d-4931-a76e-e380be4fd804" (UID: "d6893a4e-1f1d-4931-a76e-e380be4fd804"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:40:52.543000 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.542965    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6893a4e-1f1d-4931-a76e-e380be4fd804-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:52.543000 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.542993    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6893a4e-1f1d-4931-a76e-e380be4fd804-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:52.543000 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.543002    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8xzb\" (UniqueName: \"kubernetes.io/projected/d6893a4e-1f1d-4931-a76e-e380be4fd804-kube-api-access-n8xzb\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:52.561536 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.561492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:52.583025 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.582998    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f69e61-04aa-4326-b50a-03fa13787342" path="/var/lib/kubelet/pods/b7f69e61-04aa-4326-b50a-03fa13787342/volumes"
Jul 28 20:40:52.801574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:52.801536    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:53.386417 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:53.386386    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"
Jul 28 20:40:53.403041 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:53.403014    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"]
Jul 28 20:40:53.406963 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:53.406942    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ff6b3-656d46b56b-pj49g"]
Jul 28 20:40:54.584005 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:54.583971    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6893a4e-1f1d-4931-a76e-e380be4fd804" path="/var/lib/kubelet/pods/d6893a4e-1f1d-4931-a76e-e380be4fd804/volumes"
Jul 28 20:40:55.534989 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.534937    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg" podUID="093204b3-1e91-4f1e-a16e-1ed3553be129" containerName="main" containerID="cri-o://49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f" gracePeriod=45
Jul 28 20:40:55.784240 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.784217    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:40:55.873660 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.873578    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpljj\" (UniqueName: \"kubernetes.io/projected/093204b3-1e91-4f1e-a16e-1ed3553be129-kube-api-access-bpljj\") pod \"093204b3-1e91-4f1e-a16e-1ed3553be129\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") "
Jul 28 20:40:55.873660 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.873641    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-dshm\") pod \"093204b3-1e91-4f1e-a16e-1ed3553be129\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") "
Jul 28 20:40:55.873857 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.873698    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-model-cache\") pod \"093204b3-1e91-4f1e-a16e-1ed3553be129\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") "
Jul 28 20:40:55.873857 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.873732    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093204b3-1e91-4f1e-a16e-1ed3553be129-tls-certs\") pod \"093204b3-1e91-4f1e-a16e-1ed3553be129\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") "
Jul 28 20:40:55.873857 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.873785    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-home\") pod \"093204b3-1e91-4f1e-a16e-1ed3553be129\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") "
Jul 28 20:40:55.873857 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.873820    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-tmp-dir\") pod \"093204b3-1e91-4f1e-a16e-1ed3553be129\" (UID: \"093204b3-1e91-4f1e-a16e-1ed3553be129\") "
Jul 28 20:40:55.874156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.873987    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-model-cache" (OuterVolumeSpecName: "model-cache") pod "093204b3-1e91-4f1e-a16e-1ed3553be129" (UID: "093204b3-1e91-4f1e-a16e-1ed3553be129"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:40:55.874156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.874095    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-model-cache\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:55.874156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.874087    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-home" (OuterVolumeSpecName: "home") pod "093204b3-1e91-4f1e-a16e-1ed3553be129" (UID: "093204b3-1e91-4f1e-a16e-1ed3553be129"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:40:55.874274 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.874163    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "093204b3-1e91-4f1e-a16e-1ed3553be129" (UID: "093204b3-1e91-4f1e-a16e-1ed3553be129"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:40:55.875861 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.875841    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093204b3-1e91-4f1e-a16e-1ed3553be129-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "093204b3-1e91-4f1e-a16e-1ed3553be129" (UID: "093204b3-1e91-4f1e-a16e-1ed3553be129"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:40:55.875933 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.875867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-dshm" (OuterVolumeSpecName: "dshm") pod "093204b3-1e91-4f1e-a16e-1ed3553be129" (UID: "093204b3-1e91-4f1e-a16e-1ed3553be129"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:40:55.875970 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.875949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093204b3-1e91-4f1e-a16e-1ed3553be129-kube-api-access-bpljj" (OuterVolumeSpecName: "kube-api-access-bpljj") pod "093204b3-1e91-4f1e-a16e-1ed3553be129" (UID: "093204b3-1e91-4f1e-a16e-1ed3553be129"). InnerVolumeSpecName "kube-api-access-bpljj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:40:55.974997 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.974961    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-dshm\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:55.974997 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.974989    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093204b3-1e91-4f1e-a16e-1ed3553be129-tls-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:55.974997 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.974999    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-home\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:55.974997 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.975006    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093204b3-1e91-4f1e-a16e-1ed3553be129-tmp-dir\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:55.975270 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:55.975016    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpljj\" (UniqueName: \"kubernetes.io/projected/093204b3-1e91-4f1e-a16e-1ed3553be129-kube-api-access-bpljj\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:40:56.401923 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.401890    2582 generic.go:356] "Generic (PLEG): container finished" podID="093204b3-1e91-4f1e-a16e-1ed3553be129" containerID="49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f" exitCode=0
Jul 28 20:40:56.402083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.401934    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"
Jul 28 20:40:56.402083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.401967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg" event={"ID":"093204b3-1e91-4f1e-a16e-1ed3553be129","Type":"ContainerDied","Data":"49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f"}
Jul 28 20:40:56.402083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.402007    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg" event={"ID":"093204b3-1e91-4f1e-a16e-1ed3553be129","Type":"ContainerDied","Data":"56ba192ac83cb76041b726689c6ed46e7957d70863be73b8f70284c3c7073528"}
Jul 28 20:40:56.402083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.402028    2582 scope.go:117] "RemoveContainer" containerID="49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f"
Jul 28 20:40:56.411222 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.411202    2582 scope.go:117] "RemoveContainer" containerID="49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f"
Jul 28 20:40:56.411489 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:40:56.411471    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f\": container with ID starting with 49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f not found: ID does not exist" containerID="49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f"
Jul 28 20:40:56.411554 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.411498    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f"} err="failed to get container status \"49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f\": rpc error: code = NotFound desc = could not find container \"49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f\": container with ID starting with 49e47b93fa5ef0b962d930d42a93b5aebb63e71ca31b02f211695d6bb5e7f24f not found: ID does not exist"
Jul 28 20:40:56.426500 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.426469    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"]
Jul 28 20:40:56.435461 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.435440    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7ff6b3/test-model-7ff6b3-kserve-84d7fcb8cc-v7cgg"]
Jul 28 20:40:56.583042 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:56.583009    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093204b3-1e91-4f1e-a16e-1ed3553be129" path="/var/lib/kubelet/pods/093204b3-1e91-4f1e-a16e-1ed3553be129/volumes"
Jul 28 20:40:58.679616 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:58.679580    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:58.750762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:58.750733    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:58.914156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:58.914124    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"]
Jul 28 20:40:58.914392 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:58.914369    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy" containerID="cri-o://6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46" gracePeriod=30
Jul 28 20:40:58.945891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:58.945812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:59.226988 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.226893    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:59.345979 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.345943    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"]
Jul 28 20:40:59.346375 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.346362    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f69e61-04aa-4326-b50a-03fa13787342" containerName="revoke-keys"
Jul 28 20:40:59.346375 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.346376    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f69e61-04aa-4326-b50a-03fa13787342" containerName="revoke-keys"
Jul 28 20:40:59.346474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.346391    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6893a4e-1f1d-4931-a76e-e380be4fd804" containerName="maas-api"
Jul 28 20:40:59.346474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.346397    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6893a4e-1f1d-4931-a76e-e380be4fd804" containerName="maas-api"
Jul 28 20:40:59.346474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.346405    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093204b3-1e91-4f1e-a16e-1ed3553be129" containerName="main"
Jul 28 20:40:59.346474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.346411    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="093204b3-1e91-4f1e-a16e-1ed3553be129" containerName="main"
Jul 28 20:40:59.346474 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.346468    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="093204b3-1e91-4f1e-a16e-1ed3553be129" containerName="main"
Jul 28 20:40:59.346625 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.346480    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f69e61-04aa-4326-b50a-03fa13787342" containerName="revoke-keys"
Jul 28 20:40:59.346625 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.346486    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6893a4e-1f1d-4931-a76e-e380be4fd804" containerName="maas-api"
Jul 28 20:40:59.350847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.350819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"
Jul 28 20:40:59.355214 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.355193    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\""
Jul 28 20:40:59.397508 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.397469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"]
Jul 28 20:40:59.402471 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.402442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt\" (UID: \"73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"
Jul 28 20:40:59.503472 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.503439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt\" (UID: \"73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"
Jul 28 20:40:59.504072 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.504045    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt\" (UID: \"73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"
Jul 28 20:40:59.661516 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.661477    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"
Jul 28 20:40:59.798350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:40:59.798324    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"]
Jul 28 20:40:59.801428 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:40:59.801401    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f24b0b_3ef9_414d_b661_5f4ffdfc6f1c.slice/crio-7452b97c2f7b4464e59e285e428ef5afe58221be9af12e867c6ab8858cbbea13 WatchSource:0}: Error finding container 7452b97c2f7b4464e59e285e428ef5afe58221be9af12e867c6ab8858cbbea13: Status 404 returned error can't find the container with id 7452b97c2f7b4464e59e285e428ef5afe58221be9af12e867c6ab8858cbbea13
Jul 28 20:41:00.425008 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:00.424901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt" event={"ID":"73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c","Type":"ContainerStarted","Data":"eb0cfc7c44b5a2de6d15f93b07f2c85eb77d09bd6a5efbce8af6ad96f403017e"}
Jul 28 20:41:00.425008 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:00.424981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt" event={"ID":"73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c","Type":"ContainerStarted","Data":"7452b97c2f7b4464e59e285e428ef5afe58221be9af12e867c6ab8858cbbea13"}
Jul 28 20:41:00.446192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:00.446146    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt" podStartSLOduration=1.44613099 podStartE2EDuration="1.44613099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:00.446074211 +0000 UTC m=+2510.300385278" watchObservedRunningTime="2026-07-28 20:41:00.44613099 +0000 UTC m=+2510.300442056"
Jul 28 20:41:01.431723 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:01.431689    2582 generic.go:356] "Generic (PLEG): container finished" podID="73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c" containerID="eb0cfc7c44b5a2de6d15f93b07f2c85eb77d09bd6a5efbce8af6ad96f403017e" exitCode=0
Jul 28 20:41:01.432172 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:01.431787    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt" event={"ID":"73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c","Type":"ContainerDied","Data":"eb0cfc7c44b5a2de6d15f93b07f2c85eb77d09bd6a5efbce8af6ad96f403017e"}
Jul 28 20:41:02.582734 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:02.582706    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"
Jul 28 20:41:02.631162 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:02.631120    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c-maas-api-service-ca\") pod \"73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c\" (UID: \"73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c\") "
Jul 28 20:41:02.631486 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:02.631464    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c" (UID: "73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:41:02.732416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:02.732327    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c-maas-api-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:03.443251 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:03.443218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt" event={"ID":"73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c","Type":"ContainerDied","Data":"7452b97c2f7b4464e59e285e428ef5afe58221be9af12e867c6ab8858cbbea13"}
Jul 28 20:41:03.443447 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:03.443265    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7452b97c2f7b4464e59e285e428ef5afe58221be9af12e867c6ab8858cbbea13"
Jul 28 20:41:03.443447 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:03.443233    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"
Jul 28 20:41:04.171254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.171226    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:41:04.245942 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.245913    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-certs\") pod \"e20c43c4-dfea-499d-9d04-28506a5bf109\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") "
Jul 28 20:41:04.245942 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.245945    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-socket\") pod \"e20c43c4-dfea-499d-9d04-28506a5bf109\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") "
Jul 28 20:41:04.246183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.245987    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-credential-socket\") pod \"e20c43c4-dfea-499d-9d04-28506a5bf109\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") "
Jul 28 20:41:04.246183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246015    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20c43c4-dfea-499d-9d04-28506a5bf109-istiod-ca-cert\") pod \"e20c43c4-dfea-499d-9d04-28506a5bf109\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") "
Jul 28 20:41:04.246183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246048    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-envoy\") pod \"e20c43c4-dfea-499d-9d04-28506a5bf109\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") "
Jul 28 20:41:04.246183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246089    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gffmz\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-kube-api-access-gffmz\") pod \"e20c43c4-dfea-499d-9d04-28506a5bf109\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") "
Jul 28 20:41:04.246183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246114    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-podinfo\") pod \"e20c43c4-dfea-499d-9d04-28506a5bf109\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") "
Jul 28 20:41:04.246183 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246161    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-data\") pod \"e20c43c4-dfea-499d-9d04-28506a5bf109\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") "
Jul 28 20:41:04.246462 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246226    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-token\") pod \"e20c43c4-dfea-499d-9d04-28506a5bf109\" (UID: \"e20c43c4-dfea-499d-9d04-28506a5bf109\") "
Jul 28 20:41:04.246462 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246236    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e20c43c4-dfea-499d-9d04-28506a5bf109" (UID: "e20c43c4-dfea-499d-9d04-28506a5bf109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:04.246462 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e20c43c4-dfea-499d-9d04-28506a5bf109" (UID: "e20c43c4-dfea-499d-9d04-28506a5bf109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:04.246462 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246349    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e20c43c4-dfea-499d-9d04-28506a5bf109" (UID: "e20c43c4-dfea-499d-9d04-28506a5bf109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:04.246607 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246563    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20c43c4-dfea-499d-9d04-28506a5bf109-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e20c43c4-dfea-499d-9d04-28506a5bf109" (UID: "e20c43c4-dfea-499d-9d04-28506a5bf109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:41:04.246607 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246573    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:04.246607 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246596    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-workload-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:04.246764 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246612    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-credential-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:04.246895 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.246875    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-data" (OuterVolumeSpecName: "istio-data") pod "e20c43c4-dfea-499d-9d04-28506a5bf109" (UID: "e20c43c4-dfea-499d-9d04-28506a5bf109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:04.248445 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.248423    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e20c43c4-dfea-499d-9d04-28506a5bf109" (UID: "e20c43c4-dfea-499d-9d04-28506a5bf109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:04.248445 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.248424    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e20c43c4-dfea-499d-9d04-28506a5bf109" (UID: "e20c43c4-dfea-499d-9d04-28506a5bf109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:41:04.248821 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.248804    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-kube-api-access-gffmz" (OuterVolumeSpecName: "kube-api-access-gffmz") pod "e20c43c4-dfea-499d-9d04-28506a5bf109" (UID: "e20c43c4-dfea-499d-9d04-28506a5bf109"). InnerVolumeSpecName "kube-api-access-gffmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:41:04.248877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.248860    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-token" (OuterVolumeSpecName: "istio-token") pod "e20c43c4-dfea-499d-9d04-28506a5bf109" (UID: "e20c43c4-dfea-499d-9d04-28506a5bf109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:41:04.347891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.347859    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-data\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:04.347891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.347886    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:04.347891 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.347895    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20c43c4-dfea-499d-9d04-28506a5bf109-istiod-ca-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:04.348204 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.347905    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-envoy\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:04.348204 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.347919    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gffmz\" (UniqueName: \"kubernetes.io/projected/e20c43c4-dfea-499d-9d04-28506a5bf109-kube-api-access-gffmz\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:04.348204 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.347932    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20c43c4-dfea-499d-9d04-28506a5bf109-istio-podinfo\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:04.451069 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.451030    2582 generic.go:356] "Generic (PLEG): container finished" podID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerID="6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46" exitCode=0
Jul 28 20:41:04.451314 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.451080    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"
Jul 28 20:41:04.451314 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.451112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" event={"ID":"e20c43c4-dfea-499d-9d04-28506a5bf109","Type":"ContainerDied","Data":"6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46"}
Jul 28 20:41:04.451314 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.451162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g" event={"ID":"e20c43c4-dfea-499d-9d04-28506a5bf109","Type":"ContainerDied","Data":"18d0d4ccab31711d7e472d3107448679d4005ce51e112c9d93ab76c860efa595"}
Jul 28 20:41:04.451314 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.451185    2582 scope.go:117] "RemoveContainer" containerID="6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46"
Jul 28 20:41:04.460825 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.460806    2582 scope.go:117] "RemoveContainer" containerID="dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c"
Jul 28 20:41:04.469503 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.469485    2582 scope.go:117] "RemoveContainer" containerID="6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46"
Jul 28 20:41:04.469765 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:41:04.469746    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46\": container with ID starting with 6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46 not found: ID does not exist" containerID="6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46"
Jul 28 20:41:04.469847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.469776    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46"} err="failed to get container status \"6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46\": rpc error: code = NotFound desc = could not find container \"6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46\": container with ID starting with 6dfb3c660e57c980af783cd4e7f61db1208ea0c9c9105170c4ac2ea6b48dec46 not found: ID does not exist"
Jul 28 20:41:04.469847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.469800    2582 scope.go:117] "RemoveContainer" containerID="dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c"
Jul 28 20:41:04.470039 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:41:04.470018    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c\": container with ID starting with dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c not found: ID does not exist" containerID="dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c"
Jul 28 20:41:04.470080 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.470045    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c"} err="failed to get container status \"dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c\": rpc error: code = NotFound desc = could not find container \"dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c\": container with ID starting with dd6c0e6995ea1738ce24d4f336dca112971fbca40e4ac8e435b2ed5f60ea892c not found: ID does not exist"
Jul 28 20:41:04.475904 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.475883    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"]
Jul 28 20:41:04.480523 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.480500    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7ff6b3-openshift-default-56dc96c4c8-dcc2g"]
Jul 28 20:41:04.583324 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:04.583228    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" path="/var/lib/kubelet/pods/e20c43c4-dfea-499d-9d04-28506a5bf109/volumes"
Jul 28 20:41:09.852322 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:09.852253    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"]
Jul 28 20:41:09.852744 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:09.852539    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g" podUID="569312a2-1ab6-4fc5-8007-f921b593adfa" containerName="maas-api" containerID="cri-o://a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7" gracePeriod=30
Jul 28 20:41:09.879585 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:09.879551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:09.982178 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:09.982145    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:10.074235 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.074166    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"]
Jul 28 20:41:10.076919 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.076890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:10.077377 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.077355    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d38c0f-f2tvt"]
Jul 28 20:41:10.103068 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.103006    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:41:10.147185 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.147142    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:10.200133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.200099    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569312a2-1ab6-4fc5-8007-f921b593adfa-openshift-service-ca\") pod \"569312a2-1ab6-4fc5-8007-f921b593adfa\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") "
Jul 28 20:41:10.200311 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.200150    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569312a2-1ab6-4fc5-8007-f921b593adfa-maas-api-tls\") pod \"569312a2-1ab6-4fc5-8007-f921b593adfa\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") "
Jul 28 20:41:10.200311 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.200212    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftzd2\" (UniqueName: \"kubernetes.io/projected/569312a2-1ab6-4fc5-8007-f921b593adfa-kube-api-access-ftzd2\") pod \"569312a2-1ab6-4fc5-8007-f921b593adfa\" (UID: \"569312a2-1ab6-4fc5-8007-f921b593adfa\") "
Jul 28 20:41:10.200469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.200441    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569312a2-1ab6-4fc5-8007-f921b593adfa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "569312a2-1ab6-4fc5-8007-f921b593adfa" (UID: "569312a2-1ab6-4fc5-8007-f921b593adfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:41:10.202254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.202225    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569312a2-1ab6-4fc5-8007-f921b593adfa-kube-api-access-ftzd2" (OuterVolumeSpecName: "kube-api-access-ftzd2") pod "569312a2-1ab6-4fc5-8007-f921b593adfa" (UID: "569312a2-1ab6-4fc5-8007-f921b593adfa"). InnerVolumeSpecName "kube-api-access-ftzd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:41:10.202374 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.202259    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569312a2-1ab6-4fc5-8007-f921b593adfa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "569312a2-1ab6-4fc5-8007-f921b593adfa" (UID: "569312a2-1ab6-4fc5-8007-f921b593adfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:41:10.301009 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.300972    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569312a2-1ab6-4fc5-8007-f921b593adfa-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:10.301009 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.301003    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569312a2-1ab6-4fc5-8007-f921b593adfa-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:10.301009 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.301012    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftzd2\" (UniqueName: \"kubernetes.io/projected/569312a2-1ab6-4fc5-8007-f921b593adfa-kube-api-access-ftzd2\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:41:10.479400 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.479302    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:10.483887 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.483864    2582 generic.go:356] "Generic (PLEG): container finished" podID="569312a2-1ab6-4fc5-8007-f921b593adfa" containerID="a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7" exitCode=0
Jul 28 20:41:10.483999 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.483911    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"
Jul 28 20:41:10.483999 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.483945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g" event={"ID":"569312a2-1ab6-4fc5-8007-f921b593adfa","Type":"ContainerDied","Data":"a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7"}
Jul 28 20:41:10.484081 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.483997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g" event={"ID":"569312a2-1ab6-4fc5-8007-f921b593adfa","Type":"ContainerDied","Data":"2d2f3d6bf67b5fe2ceb27af37a530821f95d18ea356d8f7b689a68b8be7235b3"}
Jul 28 20:41:10.484081 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.484025    2582 scope.go:117] "RemoveContainer" containerID="a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7"
Jul 28 20:41:10.493163 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.493144    2582 scope.go:117] "RemoveContainer" containerID="a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7"
Jul 28 20:41:10.493445 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:41:10.493425    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7\": container with ID starting with a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7 not found: ID does not exist" containerID="a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7"
Jul 28 20:41:10.493501 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.493455    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7"} err="failed to get container status \"a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7\": rpc error: code = NotFound desc = could not find container \"a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7\": container with ID starting with a3326d6e1346e34a4783a7a5207dc43475a5e3db847954b54d3d0de017ab74d7 not found: ID does not exist"
Jul 28 20:41:10.515397 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.515377    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"]
Jul 28 20:41:10.535488 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.535460    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d38c0f-675cb99489-9jl6g"]
Jul 28 20:41:10.583252 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.583224    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569312a2-1ab6-4fc5-8007-f921b593adfa" path="/var/lib/kubelet/pods/569312a2-1ab6-4fc5-8007-f921b593adfa/volumes"
Jul 28 20:41:10.583625 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:10.583611    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c" path="/var/lib/kubelet/pods/73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c/volumes"
Jul 28 20:41:16.532244 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:16.532201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:16.643875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:16.643843    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:16.822366 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:16.822254    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:17.047658 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.047624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:17.398613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.398579    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:17.698843 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.698761    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"]
Jul 28 20:41:17.699192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699113    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569312a2-1ab6-4fc5-8007-f921b593adfa" containerName="maas-api"
Jul 28 20:41:17.699192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699123    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="569312a2-1ab6-4fc5-8007-f921b593adfa" containerName="maas-api"
Jul 28 20:41:17.699192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699136    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy"
Jul 28 20:41:17.699192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699142    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy"
Jul 28 20:41:17.699192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699156    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c" containerName="revoke-keys"
Jul 28 20:41:17.699192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699161    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c" containerName="revoke-keys"
Jul 28 20:41:17.699192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699169    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy"
Jul 28 20:41:17.699192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699174    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy"
Jul 28 20:41:17.699463 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699229    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f24b0b-3ef9-414d-b661-5f4ffdfc6f1c" containerName="revoke-keys"
Jul 28 20:41:17.699463 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699237    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy"
Jul 28 20:41:17.699463 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699243    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="569312a2-1ab6-4fc5-8007-f921b593adfa" containerName="maas-api"
Jul 28 20:41:17.699463 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.699249    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20c43c4-dfea-499d-9d04-28506a5bf109" containerName="istio-proxy"
Jul 28 20:41:17.702372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.702352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.706932 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.706915    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c55858-openshift-default-dockercfg-rgdhm\""
Jul 28 20:41:17.722088 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.722062    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"]
Jul 28 20:41:17.734898 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.734856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:17.866163 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.866133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-data\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.866163 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.866178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqb2m\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-kube-api-access-pqb2m\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.866411 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.866313    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istiod-ca-cert\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.866411 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.866363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-token\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.866411 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.866402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-podinfo\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.866536 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.866441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-certs\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.866536 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.866477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-envoy\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.866605 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.866573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-credential-socket\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.866640 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.866620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-socket\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967167 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-token\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967167 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-podinfo\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967167 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-certs\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-envoy\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-credential-socket\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-socket\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-data\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqb2m\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-kube-api-access-pqb2m\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967498 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istiod-ca-cert\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967805 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-certs\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967805 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-credential-socket\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967805 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-socket\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.967936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.967917    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-data\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.968404 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.968380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istiod-ca-cert\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.969657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.969639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-envoy\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.970013 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.969996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-podinfo\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.976134 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.976109    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-token\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:17.976433 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:17.976408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqb2m\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-kube-api-access-pqb2m\") pod \"e2e-api-a-c55858-openshift-default-6776567764-fk9k4\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") " pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:18.012203 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:18.012173    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:18.151659 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:18.151624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:18.170131 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:18.169998    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"]
Jul 28 20:41:18.172844 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:41:18.172815    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcab854b_e4b3_4e38_9b0b_70e0c5d67baf.slice/crio-fc7460aceaf5f8bd9efd4df05250dca3105ea99ff3eaf4a71f5bfd475dfd21e5 WatchSource:0}: Error finding container fc7460aceaf5f8bd9efd4df05250dca3105ea99ff3eaf4a71f5bfd475dfd21e5: Status 404 returned error can't find the container with id fc7460aceaf5f8bd9efd4df05250dca3105ea99ff3eaf4a71f5bfd475dfd21e5
Jul 28 20:41:18.174982 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:18.174950    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:41:18.175063 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:18.175018    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:41:18.175063 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:18.175049    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:41:18.529885 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:18.529796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" event={"ID":"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf","Type":"ContainerStarted","Data":"f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd"}
Jul 28 20:41:18.530030 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:18.529892    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" event={"ID":"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf","Type":"ContainerStarted","Data":"fc7460aceaf5f8bd9efd4df05250dca3105ea99ff3eaf4a71f5bfd475dfd21e5"}
Jul 28 20:41:19.013372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:19.013341    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:19.018517 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:19.018492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:19.054307 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:19.054252    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" podStartSLOduration=2.054235938 podStartE2EDuration="2.054235938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:18.567404432 +0000 UTC m=+2528.421715501" watchObservedRunningTime="2026-07-28 20:41:19.054235938 +0000 UTC m=+2528.908547003"
Jul 28 20:41:19.535101 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:19.535063    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:19.536050 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:19.536026    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:20.609458 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:20.609419    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:20.760396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:20.760353    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:20.891983 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:20.891883    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"]
Jul 28 20:41:20.896025 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:20.896008    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:20.900600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:20.900577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c55858-serving-cert\""
Jul 28 20:41:20.941080 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:20.941043    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"]
Jul 28 20:41:20.999797 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:20.999757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-c55858-59b47456b9-dc7rm\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:20.999797 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:20.999800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-maas-api-tls\") pod \"maas-api-e2e-api-a-c55858-59b47456b9-dc7rm\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:21.000059 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:20.999950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpxt\" (UniqueName: \"kubernetes.io/projected/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-kube-api-access-kqpxt\") pod \"maas-api-e2e-api-a-c55858-59b47456b9-dc7rm\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:21.012797 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.012759    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:21.100727 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.100692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpxt\" (UniqueName: \"kubernetes.io/projected/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-kube-api-access-kqpxt\") pod \"maas-api-e2e-api-a-c55858-59b47456b9-dc7rm\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:21.100912 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.100752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-c55858-59b47456b9-dc7rm\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:21.100912 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.100778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-maas-api-tls\") pod \"maas-api-e2e-api-a-c55858-59b47456b9-dc7rm\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:21.101478 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.101455    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-c55858-59b47456b9-dc7rm\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:21.103227 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.103206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-maas-api-tls\") pod \"maas-api-e2e-api-a-c55858-59b47456b9-dc7rm\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:21.113624 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.113600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpxt\" (UniqueName: \"kubernetes.io/projected/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-kube-api-access-kqpxt\") pod \"maas-api-e2e-api-a-c55858-59b47456b9-dc7rm\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:21.169053 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.168948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:21.207455 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.207424    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:21.370177 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.370147    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"]
Jul 28 20:41:21.373204 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:41:21.373171    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a311f0_2032_4073_aa8c_4845a3c2a0a5.slice/crio-a933b324b6c1ca6b118fe7dba525164b0c9d343c607c71e6a4258c3e1aa26ce8 WatchSource:0}: Error finding container a933b324b6c1ca6b118fe7dba525164b0c9d343c607c71e6a4258c3e1aa26ce8: Status 404 returned error can't find the container with id a933b324b6c1ca6b118fe7dba525164b0c9d343c607c71e6a4258c3e1aa26ce8
Jul 28 20:41:21.374543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.374520    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:41:21.546105 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:21.546071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm" event={"ID":"b0a311f0-2032-4073-aa8c-4845a3c2a0a5","Type":"ContainerStarted","Data":"a933b324b6c1ca6b118fe7dba525164b0c9d343c607c71e6a4258c3e1aa26ce8"}
Jul 28 20:41:22.158415 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:22.158335    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:22.310381 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:22.310347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:22.552345 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:22.552307    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm" event={"ID":"b0a311f0-2032-4073-aa8c-4845a3c2a0a5","Type":"ContainerStarted","Data":"34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5"}
Jul 28 20:41:22.552544 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:22.552371    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:28.563522 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:28.563488    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:41:28.591124 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:28.591077    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm" podStartSLOduration=8.011841759 podStartE2EDuration="8.591062326s" totalImagesPullingTime="579.220567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:22.588132589 +0000 UTC m=+2532.442443676" watchObservedRunningTime="2026-07-28 20:41:28.591062326 +0000 UTC m=+2538.445373415"
Jul 28 20:41:33.607218 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:33.607188    2582 generic.go:356] "Generic (PLEG): container finished" podID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerID="f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd" exitCode=0
Jul 28 20:41:33.607676 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:33.607255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" event={"ID":"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf","Type":"ContainerDied","Data":"f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd"}
Jul 28 20:41:33.607676 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:33.607664    2582 scope.go:117] "RemoveContainer" containerID="f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd"
Jul 28 20:41:33.608646 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:33.608621    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:41:33.608716 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:33.608685    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:41:33.608754 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:33.608717    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:41:34.536153 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:34.536100    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:34.614152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:34.614117    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" event={"ID":"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf","Type":"ContainerStarted","Data":"4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a"}
Jul 28 20:41:35.012484 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:35.012455    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:35.013659 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:35.013637    2582 patch_prober.go:28] interesting pod/e2e-api-a-c55858-openshift-default-6776567764-fk9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Jul 28 20:41:35.013791 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:35.013680    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Jul 28 20:41:36.013496 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:36.013461    2582 patch_prober.go:28] interesting pod/e2e-api-a-c55858-openshift-default-6776567764-fk9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Jul 28 20:41:36.013905 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:36.013516    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Jul 28 20:41:37.013040 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:37.012999    2582 patch_prober.go:28] interesting pod/e2e-api-a-c55858-openshift-default-6776567764-fk9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Jul 28 20:41:37.013241 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:37.013061    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Jul 28 20:41:38.012833 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:38.012796    2582 patch_prober.go:28] interesting pod/e2e-api-a-c55858-openshift-default-6776567764-fk9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Jul 28 20:41:38.013235 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:38.012866    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Jul 28 20:41:39.016530 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:39.016497    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:39.016917 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:39.016857    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:39.017647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:39.017628    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:41:45.586820 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:45.586786    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:46.875457 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:46.875417    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:49.942792 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:49.942751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:50.040995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.040949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:50.230321 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.230205    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"]
Jul 28 20:41:50.233841 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.233815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.237690 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.237663    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2ed465-serving-cert\""
Jul 28 20:41:50.251485 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.251457    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"]
Jul 28 20:41:50.291902 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.291856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:50.383844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.383801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnplm\" (UniqueName: \"kubernetes.io/projected/b48bd01a-b2bf-4032-bef7-2280fde4ca01-kube-api-access-xnplm\") pod \"maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.383844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.383847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b48bd01a-b2bf-4032-bef7-2280fde4ca01-maas-api-tls\") pod \"maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.384064 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.383910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b48bd01a-b2bf-4032-bef7-2280fde4ca01-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.485296 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.485263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnplm\" (UniqueName: \"kubernetes.io/projected/b48bd01a-b2bf-4032-bef7-2280fde4ca01-kube-api-access-xnplm\") pod \"maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.485453 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.485323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b48bd01a-b2bf-4032-bef7-2280fde4ca01-maas-api-tls\") pod \"maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.485453 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.485372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b48bd01a-b2bf-4032-bef7-2280fde4ca01-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.486484 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.486442    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b48bd01a-b2bf-4032-bef7-2280fde4ca01-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.491570 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.491541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b48bd01a-b2bf-4032-bef7-2280fde4ca01-maas-api-tls\") pod \"maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.499630 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.499600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnplm\" (UniqueName: \"kubernetes.io/projected/b48bd01a-b2bf-4032-bef7-2280fde4ca01-kube-api-access-xnplm\") pod \"maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.525332 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.525283    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:50.544688 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.544660    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:50.705049 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:50.705019    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"]
Jul 28 20:41:50.708211 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:41:50.708185    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48bd01a_b2bf_4032_bef7_2280fde4ca01.slice/crio-79a7637420e8ab2468f0253fafcddf1cdf8727cd73719a41a11de4db29cdbac4 WatchSource:0}: Error finding container 79a7637420e8ab2468f0253fafcddf1cdf8727cd73719a41a11de4db29cdbac4: Status 404 returned error can't find the container with id 79a7637420e8ab2468f0253fafcddf1cdf8727cd73719a41a11de4db29cdbac4
Jul 28 20:41:51.066845 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.066813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:51.220523 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.220480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:51.268987 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.268949    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"]
Jul 28 20:41:51.272544 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.272522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:41:51.275760 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.275731    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2ed465-dockercfg-pqsp5\""
Jul 28 20:41:51.276209 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.276187    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2ed465\""
Jul 28 20:41:51.288196 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.288167    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"]
Jul 28 20:41:51.392567 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.392475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e61fd-a930-43ec-8564-20ac67f72163-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9\" (UID: \"504e61fd-a930-43ec-8564-20ac67f72163\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:41:51.392741 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.392620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxfb\" (UniqueName: \"kubernetes.io/projected/504e61fd-a930-43ec-8564-20ac67f72163-kube-api-access-6xxfb\") pod \"payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9\" (UID: \"504e61fd-a930-43ec-8564-20ac67f72163\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:41:51.493976 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.493934    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxfb\" (UniqueName: \"kubernetes.io/projected/504e61fd-a930-43ec-8564-20ac67f72163-kube-api-access-6xxfb\") pod \"payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9\" (UID: \"504e61fd-a930-43ec-8564-20ac67f72163\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:41:51.494220 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.494035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e61fd-a930-43ec-8564-20ac67f72163-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9\" (UID: \"504e61fd-a930-43ec-8564-20ac67f72163\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:41:51.494816 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.494789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e61fd-a930-43ec-8564-20ac67f72163-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9\" (UID: \"504e61fd-a930-43ec-8564-20ac67f72163\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:41:51.504825 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.504800    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxfb\" (UniqueName: \"kubernetes.io/projected/504e61fd-a930-43ec-8564-20ac67f72163-kube-api-access-6xxfb\") pod \"payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9\" (UID: \"504e61fd-a930-43ec-8564-20ac67f72163\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:41:51.583657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.583622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:41:51.705211 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.705173    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc" event={"ID":"b48bd01a-b2bf-4032-bef7-2280fde4ca01","Type":"ContainerStarted","Data":"38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007"}
Jul 28 20:41:51.705413 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.705231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc" event={"ID":"b48bd01a-b2bf-4032-bef7-2280fde4ca01","Type":"ContainerStarted","Data":"79a7637420e8ab2468f0253fafcddf1cdf8727cd73719a41a11de4db29cdbac4"}
Jul 28 20:41:51.705413 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.705285    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:41:51.742196 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.742163    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"]
Jul 28 20:41:51.747457 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:41:51.747419    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504e61fd_a930_43ec_8564_20ac67f72163.slice/crio-1df481c024a10b46dfd3992712a29b56f0bfbce817edbcb7eab2ecab9d333b60 WatchSource:0}: Error finding container 1df481c024a10b46dfd3992712a29b56f0bfbce817edbcb7eab2ecab9d333b60: Status 404 returned error can't find the container with id 1df481c024a10b46dfd3992712a29b56f0bfbce817edbcb7eab2ecab9d333b60
Jul 28 20:41:51.774135 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:51.774083    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc" podStartSLOduration=1.352121099 podStartE2EDuration="1.77406683s" totalImagesPullingTime="421.945731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:51.773175692 +0000 UTC m=+2561.627486758" watchObservedRunningTime="2026-07-28 20:41:51.77406683 +0000 UTC m=+2561.628377895"
Jul 28 20:41:52.712073 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:52.712035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9" event={"ID":"504e61fd-a930-43ec-8564-20ac67f72163","Type":"ContainerStarted","Data":"f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f"}
Jul 28 20:41:52.712536 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:52.712097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9" event={"ID":"504e61fd-a930-43ec-8564-20ac67f72163","Type":"ContainerStarted","Data":"1df481c024a10b46dfd3992712a29b56f0bfbce817edbcb7eab2ecab9d333b60"}
Jul 28 20:41:52.712536 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:52.712282    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:41:52.741526 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:52.741468    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9" podStartSLOduration=1.741452877 podStartE2EDuration="1.741452877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:52.740987427 +0000 UTC m=+2562.595298493" watchObservedRunningTime="2026-07-28 20:41:52.741452877 +0000 UTC m=+2562.595764020"
Jul 28 20:41:57.718087 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:41:57.718058    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:42:01.440113 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:01.440080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:03.719432 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:03.719396    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:42:10.907888 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:10.907856    2582 scope.go:117] "RemoveContainer" containerID="25fe125e67d761b4078343286c3dd62125524a9e26af5292b75818bf25833263"
Jul 28 20:42:14.892044 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:14.892009    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:17.966957 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:17.966905    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"]
Jul 28 20:42:17.971146 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:17.971121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"
Jul 28 20:42:17.974598 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:17.974576    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\""
Jul 28 20:42:17.980238 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:17.980196    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"]
Jul 28 20:42:18.034950 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:18.034916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76eef1a5-e577-4004-b838-dc19375fce1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c55858-x742d\" (UID: \"76eef1a5-e577-4004-b838-dc19375fce1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"
Jul 28 20:42:18.136318 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:18.136256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76eef1a5-e577-4004-b838-dc19375fce1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c55858-x742d\" (UID: \"76eef1a5-e577-4004-b838-dc19375fce1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"
Jul 28 20:42:18.136952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:18.136931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76eef1a5-e577-4004-b838-dc19375fce1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c55858-x742d\" (UID: \"76eef1a5-e577-4004-b838-dc19375fce1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"
Jul 28 20:42:18.284519 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:18.284478    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"
Jul 28 20:42:18.423063 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:18.422973    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"]
Jul 28 20:42:18.426215 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:42:18.426183    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76eef1a5_e577_4004_b838_dc19375fce1d.slice/crio-47377edc5f6eb12b5de780358c4a7980c2a109afce41fccfe1b18dd6354ca475 WatchSource:0}: Error finding container 47377edc5f6eb12b5de780358c4a7980c2a109afce41fccfe1b18dd6354ca475: Status 404 returned error can't find the container with id 47377edc5f6eb12b5de780358c4a7980c2a109afce41fccfe1b18dd6354ca475
Jul 28 20:42:18.848562 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:18.848528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d" event={"ID":"76eef1a5-e577-4004-b838-dc19375fce1d","Type":"ContainerStarted","Data":"1cc562d933c47754786447502471bbd72b8a73cc366f399ea4c67f80b1f620d2"}
Jul 28 20:42:18.848729 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:18.848580    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d" event={"ID":"76eef1a5-e577-4004-b838-dc19375fce1d","Type":"ContainerStarted","Data":"47377edc5f6eb12b5de780358c4a7980c2a109afce41fccfe1b18dd6354ca475"}
Jul 28 20:42:18.884602 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:18.884481    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d" podStartSLOduration=1.884458207 podStartE2EDuration="1.884458207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:42:18.880693629 +0000 UTC m=+2588.735004696" watchObservedRunningTime="2026-07-28 20:42:18.884458207 +0000 UTC m=+2588.738769279"
Jul 28 20:42:19.854764 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:19.854732    2582 generic.go:356] "Generic (PLEG): container finished" podID="76eef1a5-e577-4004-b838-dc19375fce1d" containerID="1cc562d933c47754786447502471bbd72b8a73cc366f399ea4c67f80b1f620d2" exitCode=0
Jul 28 20:42:19.855162 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:19.854822    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d" event={"ID":"76eef1a5-e577-4004-b838-dc19375fce1d","Type":"ContainerDied","Data":"1cc562d933c47754786447502471bbd72b8a73cc366f399ea4c67f80b1f620d2"}
Jul 28 20:42:21.000645 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:21.000615    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"
Jul 28 20:42:21.061936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:21.061903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76eef1a5-e577-4004-b838-dc19375fce1d-maas-api-service-ca\") pod \"76eef1a5-e577-4004-b838-dc19375fce1d\" (UID: \"76eef1a5-e577-4004-b838-dc19375fce1d\") "
Jul 28 20:42:21.062254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:21.062231    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76eef1a5-e577-4004-b838-dc19375fce1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76eef1a5-e577-4004-b838-dc19375fce1d" (UID: "76eef1a5-e577-4004-b838-dc19375fce1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:42:21.162800 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:21.162716    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76eef1a5-e577-4004-b838-dc19375fce1d-maas-api-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:21.866774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:21.866748    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"
Jul 28 20:42:21.866774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:21.866764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d" event={"ID":"76eef1a5-e577-4004-b838-dc19375fce1d","Type":"ContainerDied","Data":"47377edc5f6eb12b5de780358c4a7980c2a109afce41fccfe1b18dd6354ca475"}
Jul 28 20:42:21.866998 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:21.866809    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47377edc5f6eb12b5de780358c4a7980c2a109afce41fccfe1b18dd6354ca475"
Jul 28 20:42:33.561922 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.561823    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"]
Jul 28 20:42:33.562429 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.562067    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm" podUID="b0a311f0-2032-4073-aa8c-4845a3c2a0a5" containerName="maas-api" containerID="cri-o://34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5" gracePeriod=30
Jul 28 20:42:33.641943 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.641906    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:33.746892 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.746852    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:33.826563 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.826481    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:42:33.872781 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.872735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:33.877600 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.877574    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqpxt\" (UniqueName: \"kubernetes.io/projected/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-kube-api-access-kqpxt\") pod \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") "
Jul 28 20:42:33.877750 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.877623    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-openshift-service-ca\") pod \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") "
Jul 28 20:42:33.877750 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.877725    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-maas-api-tls\") pod \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\" (UID: \"b0a311f0-2032-4073-aa8c-4845a3c2a0a5\") "
Jul 28 20:42:33.878002 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.877978    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0a311f0-2032-4073-aa8c-4845a3c2a0a5" (UID: "b0a311f0-2032-4073-aa8c-4845a3c2a0a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:42:33.879762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.879734    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0a311f0-2032-4073-aa8c-4845a3c2a0a5" (UID: "b0a311f0-2032-4073-aa8c-4845a3c2a0a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:42:33.879832 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.879797    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-kube-api-access-kqpxt" (OuterVolumeSpecName: "kube-api-access-kqpxt") pod "b0a311f0-2032-4073-aa8c-4845a3c2a0a5" (UID: "b0a311f0-2032-4073-aa8c-4845a3c2a0a5"). InnerVolumeSpecName "kube-api-access-kqpxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:33.889841 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.889814    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"]
Jul 28 20:42:33.896256 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.896232    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c55858-x742d"]
Jul 28 20:42:33.927667 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.927631    2582 generic.go:356] "Generic (PLEG): container finished" podID="b0a311f0-2032-4073-aa8c-4845a3c2a0a5" containerID="34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5" exitCode=0
Jul 28 20:42:33.927846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.927697    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"
Jul 28 20:42:33.927846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.927714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm" event={"ID":"b0a311f0-2032-4073-aa8c-4845a3c2a0a5","Type":"ContainerDied","Data":"34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5"}
Jul 28 20:42:33.927846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.927768    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm" event={"ID":"b0a311f0-2032-4073-aa8c-4845a3c2a0a5","Type":"ContainerDied","Data":"a933b324b6c1ca6b118fe7dba525164b0c9d343c607c71e6a4258c3e1aa26ce8"}
Jul 28 20:42:33.927846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.927791    2582 scope.go:117] "RemoveContainer" containerID="34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5"
Jul 28 20:42:33.937812 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.937790    2582 scope.go:117] "RemoveContainer" containerID="34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5"
Jul 28 20:42:33.938071 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:42:33.938050    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5\": container with ID starting with 34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5 not found: ID does not exist" containerID="34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5"
Jul 28 20:42:33.938125 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.938080    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5"} err="failed to get container status \"34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5\": rpc error: code = NotFound desc = could not find container \"34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5\": container with ID starting with 34189ca1d72e1a5c7b8f1b20e75e77eda04584ddde7309b7b3988cad8affc7a5 not found: ID does not exist"
Jul 28 20:42:33.970990 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.970958    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"]
Jul 28 20:42:33.979117 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.979085    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpxt\" (UniqueName: \"kubernetes.io/projected/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-kube-api-access-kqpxt\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:33.979267 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.979123    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:33.979267 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.979140    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a311f0-2032-4073-aa8c-4845a3c2a0a5-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:33.985979 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:33.985953    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c55858-59b47456b9-dc7rm"]
Jul 28 20:42:34.328565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:34.328526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:34.583538 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:34.583458    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76eef1a5-e577-4004-b838-dc19375fce1d" path="/var/lib/kubelet/pods/76eef1a5-e577-4004-b838-dc19375fce1d/volumes"
Jul 28 20:42:34.583922 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:34.583841    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a311f0-2032-4073-aa8c-4845a3c2a0a5" path="/var/lib/kubelet/pods/b0a311f0-2032-4073-aa8c-4845a3c2a0a5/volumes"
Jul 28 20:42:35.450863 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:35.450818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:35.525372 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:35.525342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:42.165317 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:42.165252    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:42.367742 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:42.367689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:42.396437 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:42.396406    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"]
Jul 28 20:42:42.396666 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:42.396641    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy" containerID="cri-o://4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a" gracePeriod=30
Jul 28 20:42:42.698192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:42.698152    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:42.872020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:42.871979    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:47.661435 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.661364    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:42:47.808864 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.808828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-token\") pod \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") "
Jul 28 20:42:47.808864 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.808866    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-podinfo\") pod \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") "
Jul 28 20:42:47.809107 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.808916    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-credential-socket\") pod \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") "
Jul 28 20:42:47.809107 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.808973    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqb2m\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-kube-api-access-pqb2m\") pod \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") "
Jul 28 20:42:47.809107 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-data\") pod \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") "
Jul 28 20:42:47.809107 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-envoy\") pod \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") "
Jul 28 20:42:47.809107 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809100    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-certs\") pod \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") "
Jul 28 20:42:47.809405 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809137    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istiod-ca-cert\") pod \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") "
Jul 28 20:42:47.809405 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809168    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-socket\") pod \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\" (UID: \"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf\") "
Jul 28 20:42:47.809405 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809253    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" (UID: "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:47.809571 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809413    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" (UID: "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:47.809571 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809503    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" (UID: "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:47.809683 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809584    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-credential-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:47.809683 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809607    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:47.809784 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809719    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" (UID: "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:42:47.809942 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.809915    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-data" (OuterVolumeSpecName: "istio-data") pod "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" (UID: "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:47.811199 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.811171    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-token" (OuterVolumeSpecName: "istio-token") pod "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" (UID: "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:47.811403 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.811229    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-kube-api-access-pqb2m" (OuterVolumeSpecName: "kube-api-access-pqb2m") pod "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" (UID: "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf"). InnerVolumeSpecName "kube-api-access-pqb2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:47.811554 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.811525    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" (UID: "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:47.811647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.811606    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" (UID: "dcab854b-e4b3-4e38-9b0b-70e0c5d67baf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:42:47.910805 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.910769    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istiod-ca-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:47.910805 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.910801    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-workload-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:47.910805 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.910811    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:47.911043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.910819    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-podinfo\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:47.911043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.910829    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqb2m\" (UniqueName: \"kubernetes.io/projected/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-kube-api-access-pqb2m\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:47.911043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.910838    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-data\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:47.911043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:47.910846    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf-istio-envoy\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:48.001943 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.001914    2582 generic.go:356] "Generic (PLEG): container finished" podID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerID="4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a" exitCode=0
Jul 28 20:42:48.002156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.001964    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"
Jul 28 20:42:48.002156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.001997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" event={"ID":"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf","Type":"ContainerDied","Data":"4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a"}
Jul 28 20:42:48.002156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.002058    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4" event={"ID":"dcab854b-e4b3-4e38-9b0b-70e0c5d67baf","Type":"ContainerDied","Data":"fc7460aceaf5f8bd9efd4df05250dca3105ea99ff3eaf4a71f5bfd475dfd21e5"}
Jul 28 20:42:48.002156 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.002089    2582 scope.go:117] "RemoveContainer" containerID="4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a"
Jul 28 20:42:48.012475 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.012452    2582 scope.go:117] "RemoveContainer" containerID="f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd"
Jul 28 20:42:48.021118 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.021098    2582 scope.go:117] "RemoveContainer" containerID="4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a"
Jul 28 20:42:48.021398 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:42:48.021378    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a\": container with ID starting with 4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a not found: ID does not exist" containerID="4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a"
Jul 28 20:42:48.021484 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.021412    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a"} err="failed to get container status \"4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a\": rpc error: code = NotFound desc = could not find container \"4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a\": container with ID starting with 4c75ee887ebf86cfa9fda16240be1d66bd8987da203dbe604587b0af23586d7a not found: ID does not exist"
Jul 28 20:42:48.021484 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.021437    2582 scope.go:117] "RemoveContainer" containerID="f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd"
Jul 28 20:42:48.021692 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:42:48.021672    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd\": container with ID starting with f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd not found: ID does not exist" containerID="f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd"
Jul 28 20:42:48.021732 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.021698    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd"} err="failed to get container status \"f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd\": rpc error: code = NotFound desc = could not find container \"f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd\": container with ID starting with f2fa1ad426a6cedaca171d501bff3c57a5cb58112cf6981fed1dfe059e5dcddd not found: ID does not exist"
Jul 28 20:42:48.035504 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.035471    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"]
Jul 28 20:42:48.040672 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.040632    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c55858-openshift-default-6776567764-fk9k4"]
Jul 28 20:42:48.583426 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:48.583395    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" path="/var/lib/kubelet/pods/dcab854b-e4b3-4e38-9b0b-70e0c5d67baf/volumes"
Jul 28 20:42:54.087821 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.087787    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"]
Jul 28 20:42:54.088283 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.088011    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc" podUID="b48bd01a-b2bf-4032-bef7-2280fde4ca01" containerName="maas-api" containerID="cri-o://38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007" gracePeriod=30
Jul 28 20:42:54.176043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.176002    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:54.279424 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.279381    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"]
Jul 28 20:42:54.279632 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.279606    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9" podUID="504e61fd-a930-43ec-8564-20ac67f72163" containerName="payload-pre-processing" containerID="cri-o://f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f" gracePeriod=30
Jul 28 20:42:54.347744 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.347634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:54.409587 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.409559    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:42:54.529718 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.529694    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:42:54.537497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.537448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:54.572248 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.572214    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b48bd01a-b2bf-4032-bef7-2280fde4ca01-openshift-service-ca\") pod \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") "
Jul 28 20:42:54.572448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.572308    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnplm\" (UniqueName: \"kubernetes.io/projected/b48bd01a-b2bf-4032-bef7-2280fde4ca01-kube-api-access-xnplm\") pod \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") "
Jul 28 20:42:54.572448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.572341    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b48bd01a-b2bf-4032-bef7-2280fde4ca01-maas-api-tls\") pod \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\" (UID: \"b48bd01a-b2bf-4032-bef7-2280fde4ca01\") "
Jul 28 20:42:54.572654 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.572629    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48bd01a-b2bf-4032-bef7-2280fde4ca01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b48bd01a-b2bf-4032-bef7-2280fde4ca01" (UID: "b48bd01a-b2bf-4032-bef7-2280fde4ca01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:42:54.574775 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.574747    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48bd01a-b2bf-4032-bef7-2280fde4ca01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b48bd01a-b2bf-4032-bef7-2280fde4ca01" (UID: "b48bd01a-b2bf-4032-bef7-2280fde4ca01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:42:54.574960 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.574934    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48bd01a-b2bf-4032-bef7-2280fde4ca01-kube-api-access-xnplm" (OuterVolumeSpecName: "kube-api-access-xnplm") pod "b48bd01a-b2bf-4032-bef7-2280fde4ca01" (UID: "b48bd01a-b2bf-4032-bef7-2280fde4ca01"). InnerVolumeSpecName "kube-api-access-xnplm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:54.670617 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.670512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:54.673517 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.673493    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e61fd-a930-43ec-8564-20ac67f72163-plugins-config-volume\") pod \"504e61fd-a930-43ec-8564-20ac67f72163\" (UID: \"504e61fd-a930-43ec-8564-20ac67f72163\") "
Jul 28 20:42:54.673696 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.673545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xxfb\" (UniqueName: \"kubernetes.io/projected/504e61fd-a930-43ec-8564-20ac67f72163-kube-api-access-6xxfb\") pod \"504e61fd-a930-43ec-8564-20ac67f72163\" (UID: \"504e61fd-a930-43ec-8564-20ac67f72163\") "
Jul 28 20:42:54.673751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.673728    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b48bd01a-b2bf-4032-bef7-2280fde4ca01-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:54.673751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.673741    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnplm\" (UniqueName: \"kubernetes.io/projected/b48bd01a-b2bf-4032-bef7-2280fde4ca01-kube-api-access-xnplm\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:54.673825 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.673751    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b48bd01a-b2bf-4032-bef7-2280fde4ca01-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:54.673880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.673858    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504e61fd-a930-43ec-8564-20ac67f72163-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "504e61fd-a930-43ec-8564-20ac67f72163" (UID: "504e61fd-a930-43ec-8564-20ac67f72163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:42:54.675564 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.675536    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504e61fd-a930-43ec-8564-20ac67f72163-kube-api-access-6xxfb" (OuterVolumeSpecName: "kube-api-access-6xxfb") pod "504e61fd-a930-43ec-8564-20ac67f72163" (UID: "504e61fd-a930-43ec-8564-20ac67f72163"). InnerVolumeSpecName "kube-api-access-6xxfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:54.774495 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.774454    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e61fd-a930-43ec-8564-20ac67f72163-plugins-config-volume\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:54.774495 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:54.774486    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xxfb\" (UniqueName: \"kubernetes.io/projected/504e61fd-a930-43ec-8564-20ac67f72163-kube-api-access-6xxfb\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:42:55.041111 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.041080    2582 generic.go:356] "Generic (PLEG): container finished" podID="504e61fd-a930-43ec-8564-20ac67f72163" containerID="f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f" exitCode=0
Jul 28 20:42:55.041329 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.041125    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"
Jul 28 20:42:55.041329 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.041159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9" event={"ID":"504e61fd-a930-43ec-8564-20ac67f72163","Type":"ContainerDied","Data":"f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f"}
Jul 28 20:42:55.041329 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.041212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9" event={"ID":"504e61fd-a930-43ec-8564-20ac67f72163","Type":"ContainerDied","Data":"1df481c024a10b46dfd3992712a29b56f0bfbce817edbcb7eab2ecab9d333b60"}
Jul 28 20:42:55.041329 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.041240    2582 scope.go:117] "RemoveContainer" containerID="f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f"
Jul 28 20:42:55.042583 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.042561    2582 generic.go:356] "Generic (PLEG): container finished" podID="b48bd01a-b2bf-4032-bef7-2280fde4ca01" containerID="38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007" exitCode=0
Jul 28 20:42:55.042703 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.042600    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"
Jul 28 20:42:55.042703 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.042617    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc" event={"ID":"b48bd01a-b2bf-4032-bef7-2280fde4ca01","Type":"ContainerDied","Data":"38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007"}
Jul 28 20:42:55.042703 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.042655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc" event={"ID":"b48bd01a-b2bf-4032-bef7-2280fde4ca01","Type":"ContainerDied","Data":"79a7637420e8ab2468f0253fafcddf1cdf8727cd73719a41a11de4db29cdbac4"}
Jul 28 20:42:55.050710 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.050694    2582 scope.go:117] "RemoveContainer" containerID="f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f"
Jul 28 20:42:55.050979 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:42:55.050962    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f\": container with ID starting with f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f not found: ID does not exist" containerID="f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f"
Jul 28 20:42:55.051059 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.050991    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f"} err="failed to get container status \"f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f\": rpc error: code = NotFound desc = could not find container \"f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f\": container with ID starting with f7ba83f58628054b1bdb28877cc08aa322f0194b74dd1c28f98404a364d42d2f not found: ID does not exist"
Jul 28 20:42:55.051059 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.051012    2582 scope.go:117] "RemoveContainer" containerID="38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007"
Jul 28 20:42:55.059383 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.059366    2582 scope.go:117] "RemoveContainer" containerID="38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007"
Jul 28 20:42:55.059650 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:42:55.059631    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007\": container with ID starting with 38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007 not found: ID does not exist" containerID="38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007"
Jul 28 20:42:55.059709 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.059660    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007"} err="failed to get container status \"38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007\": rpc error: code = NotFound desc = could not find container \"38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007\": container with ID starting with 38b9cb33f6bc20ee954138b51e91c82ff3afe9274ea1116d51ad6cb18942d007 not found: ID does not exist"
Jul 28 20:42:55.072881 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.072829    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"]
Jul 28 20:42:55.073054 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.073004    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:55.077141 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.077119    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed465-7bdfb7d7f4-22vmc"]
Jul 28 20:42:55.100085 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.100050    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"]
Jul 28 20:42:55.104919 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:55.104891    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ed465-5d67847bfb-l2nj9"]
Jul 28 20:42:56.583693 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:56.583661    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="504e61fd-a930-43ec-8564-20ac67f72163" path="/var/lib/kubelet/pods/504e61fd-a930-43ec-8564-20ac67f72163/volumes"
Jul 28 20:42:56.584064 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:42:56.584037    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b48bd01a-b2bf-4032-bef7-2280fde4ca01" path="/var/lib/kubelet/pods/b48bd01a-b2bf-4032-bef7-2280fde4ca01/volumes"
Jul 28 20:43:01.607966 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:01.607927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:01.706973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:01.706938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:01.873894 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:01.873799    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:02.108655 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.108620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:02.443366 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.443328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:02.737502 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.737470    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"]
Jul 28 20:43:02.738020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738006    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy"
Jul 28 20:43:02.738075 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738025    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy"
Jul 28 20:43:02.738075 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738044    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48bd01a-b2bf-4032-bef7-2280fde4ca01" containerName="maas-api"
Jul 28 20:43:02.738075 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738053    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48bd01a-b2bf-4032-bef7-2280fde4ca01" containerName="maas-api"
Jul 28 20:43:02.738075 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738063    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76eef1a5-e577-4004-b838-dc19375fce1d" containerName="revoke-keys"
Jul 28 20:43:02.738075 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738072    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="76eef1a5-e577-4004-b838-dc19375fce1d" containerName="revoke-keys"
Jul 28 20:43:02.738247 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738090    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy"
Jul 28 20:43:02.738247 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738098    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy"
Jul 28 20:43:02.738247 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738108    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504e61fd-a930-43ec-8564-20ac67f72163" containerName="payload-pre-processing"
Jul 28 20:43:02.738247 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738117    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="504e61fd-a930-43ec-8564-20ac67f72163" containerName="payload-pre-processing"
Jul 28 20:43:02.738247 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738132    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a311f0-2032-4073-aa8c-4845a3c2a0a5" containerName="maas-api"
Jul 28 20:43:02.738247 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738140    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a311f0-2032-4073-aa8c-4845a3c2a0a5" containerName="maas-api"
Jul 28 20:43:02.738247 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738222    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy"
Jul 28 20:43:02.738247 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738234    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a311f0-2032-4073-aa8c-4845a3c2a0a5" containerName="maas-api"
Jul 28 20:43:02.738247 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738246    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48bd01a-b2bf-4032-bef7-2280fde4ca01" containerName="maas-api"
Jul 28 20:43:02.738560 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738255    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="504e61fd-a930-43ec-8564-20ac67f72163" containerName="payload-pre-processing"
Jul 28 20:43:02.738560 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.738265    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="76eef1a5-e577-4004-b838-dc19375fce1d" containerName="revoke-keys"
Jul 28 20:43:02.743194 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.743170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.754090 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.754069    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-1b6922-openshift-default-dockercfg-7ncd2\""
Jul 28 20:43:02.766158 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.766133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"]
Jul 28 20:43:02.792312 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.792260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:02.843841 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.843800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-certs\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.843841 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.843843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-socket\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.844054 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.843868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-podinfo\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.844054 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.843914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-envoy\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.844054 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.843929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-token\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.844054 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.844008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c698e0-0686-4f6e-a3a8-a919a61eb731-istiod-ca-cert\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.844054 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.844052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsv89\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-kube-api-access-zsv89\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.844243 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.844112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-credential-socket\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.844243 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.844129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-data\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.944781 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.944746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c698e0-0686-4f6e-a3a8-a919a61eb731-istiod-ca-cert\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.944995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.944803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsv89\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-kube-api-access-zsv89\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.944995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.944845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-credential-socket\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.944995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.944865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-data\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.944995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.944892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-certs\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.944995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.944914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-socket\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.944995 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.944938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-podinfo\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.945362 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.944997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-envoy\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.945362 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.945022    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-token\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.945438 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.945380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-certs\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.945507 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.945477    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-credential-socket\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.945574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.945511    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-socket\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.945631 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.945605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-data\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.945686 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.945656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c698e0-0686-4f6e-a3a8-a919a61eb731-istiod-ca-cert\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.947497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.947479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-envoy\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.947776 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.947760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-podinfo\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.953611 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.953591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-token\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.956243 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.956213    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsv89\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-kube-api-access-zsv89\") pod \"e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") " pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:02.981910 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:02.981880    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:03.053522 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:03.053430    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:03.195064 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:03.195038    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"]
Jul 28 20:43:03.196856 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:43:03.196825    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c698e0_0686_4f6e_a3a8_a919a61eb731.slice/crio-18458fdc7531176cdd35cabf6be91c9cddd26c0da9c48c72b0437c24607fd134 WatchSource:0}: Error finding container 18458fdc7531176cdd35cabf6be91c9cddd26c0da9c48c72b0437c24607fd134: Status 404 returned error can't find the container with id 18458fdc7531176cdd35cabf6be91c9cddd26c0da9c48c72b0437c24607fd134
Jul 28 20:43:03.198970 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:03.198936    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:43:03.199026 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:03.199006    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:43:03.199063 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:03.199036    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:43:04.088768 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:04.088728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" event={"ID":"54c698e0-0686-4f6e-a3a8-a919a61eb731","Type":"ContainerStarted","Data":"0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e"}
Jul 28 20:43:04.089189 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:04.088789    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" event={"ID":"54c698e0-0686-4f6e-a3a8-a919a61eb731","Type":"ContainerStarted","Data":"18458fdc7531176cdd35cabf6be91c9cddd26c0da9c48c72b0437c24607fd134"}
Jul 28 20:43:04.121590 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:04.121540    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" podStartSLOduration=2.121523424 podStartE2EDuration="2.121523424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:43:04.120778009 +0000 UTC m=+2633.975089078" watchObservedRunningTime="2026-07-28 20:43:04.121523424 +0000 UTC m=+2633.975834489"
Jul 28 20:43:05.054256 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:05.054217    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:05.059355 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:05.059328    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:05.094601 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:05.094575    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:05.095533 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:05.095514    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:06.250370 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.250333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:06.518810 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.518714    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"]
Jul 28 20:43:06.519308 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.519270    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcab854b-e4b3-4e38-9b0b-70e0c5d67baf" containerName="istio-proxy"
Jul 28 20:43:06.521999 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.521978    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:06.524947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.524924    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-1b6922-serving-cert\""
Jul 28 20:43:06.540758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.540734    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"]
Jul 28 20:43:06.545634 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.545608    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:06.574387 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.574351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5m8s\" (UniqueName: \"kubernetes.io/projected/5e98fd13-fc76-4406-9246-6089d3e93b5f-kube-api-access-j5m8s\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:06.574559 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.574408    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e98fd13-fc76-4406-9246-6089d3e93b5f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:06.574559 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.574487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e98fd13-fc76-4406-9246-6089d3e93b5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:06.675530 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.675489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5m8s\" (UniqueName: \"kubernetes.io/projected/5e98fd13-fc76-4406-9246-6089d3e93b5f-kube-api-access-j5m8s\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:06.675729 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.675547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e98fd13-fc76-4406-9246-6089d3e93b5f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:06.675845 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.675824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e98fd13-fc76-4406-9246-6089d3e93b5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:06.676007 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:43:06.675982    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-serving-cert: secret "maas-api-e2e-shared-a-1b6922-serving-cert" not found
Jul 28 20:43:06.676089 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:43:06.676051    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e98fd13-fc76-4406-9246-6089d3e93b5f-maas-api-tls podName:5e98fd13-fc76-4406-9246-6089d3e93b5f nodeName:}" failed. No retries permitted until 2026-07-28 20:43:07.17603086 +0000 UTC m=+2637.030341913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e98fd13-fc76-4406-9246-6089d3e93b5f-maas-api-tls") pod "maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9" (UID: "5e98fd13-fc76-4406-9246-6089d3e93b5f") : secret "maas-api-e2e-shared-a-1b6922-serving-cert" not found
Jul 28 20:43:06.676470 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.676451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e98fd13-fc76-4406-9246-6089d3e93b5f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:06.705487 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.705459    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5m8s\" (UniqueName: \"kubernetes.io/projected/5e98fd13-fc76-4406-9246-6089d3e93b5f-kube-api-access-j5m8s\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:06.707810 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:06.707782    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:07.019175 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:07.019123    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:07.179888 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:07.179856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e98fd13-fc76-4406-9246-6089d3e93b5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:07.182304 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:07.182263    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e98fd13-fc76-4406-9246-6089d3e93b5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:07.433013 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:07.432907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:07.576547 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:07.576514    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:07.606680 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:07.606516    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"]
Jul 28 20:43:07.608767 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:43:07.608738    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e98fd13_fc76_4406_9246_6089d3e93b5f.slice/crio-1617fcb5bed238915418b494dfcb298323373800fc9c8d6f145db3e827923086 WatchSource:0}: Error finding container 1617fcb5bed238915418b494dfcb298323373800fc9c8d6f145db3e827923086: Status 404 returned error can't find the container with id 1617fcb5bed238915418b494dfcb298323373800fc9c8d6f145db3e827923086
Jul 28 20:43:07.710764 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:07.710681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:08.110500 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:08.110465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9" event={"ID":"5e98fd13-fc76-4406-9246-6089d3e93b5f","Type":"ContainerStarted","Data":"1617fcb5bed238915418b494dfcb298323373800fc9c8d6f145db3e827923086"}
Jul 28 20:43:09.117211 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:09.117169    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9" event={"ID":"5e98fd13-fc76-4406-9246-6089d3e93b5f","Type":"ContainerStarted","Data":"80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f"}
Jul 28 20:43:09.117670 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:09.117326    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:09.149528 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:09.149477    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9" podStartSLOduration=2.503168712 podStartE2EDuration="3.149460773s" totalImagesPullingTime="646.292061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:43:09.147924987 +0000 UTC m=+2639.002236052" watchObservedRunningTime="2026-07-28 20:43:09.149460773 +0000 UTC m=+2639.003771839"
Jul 28 20:43:13.148956 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:13.148908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:15.137223 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:15.137183    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:43:19.175950 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:19.175861    2582 generic.go:356] "Generic (PLEG): container finished" podID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerID="0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e" exitCode=0
Jul 28 20:43:19.176410 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:19.175944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" event={"ID":"54c698e0-0686-4f6e-a3a8-a919a61eb731","Type":"ContainerDied","Data":"0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e"}
Jul 28 20:43:19.176410 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:19.176333    2582 scope.go:117] "RemoveContainer" containerID="0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e"
Jul 28 20:43:19.177452 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:19.177423    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:43:19.177556 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:19.177503    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:43:19.177556 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:19.177546    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:43:20.094965 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:20.094921    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:20.182993 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:20.182960    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" event={"ID":"54c698e0-0686-4f6e-a3a8-a919a61eb731","Type":"ContainerStarted","Data":"11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b"}
Jul 28 20:43:21.054441 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:21.054404    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:21.055683 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:21.055650    2582 patch_prober.go:28] interesting pod/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Jul 28 20:43:21.055829 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:21.055729    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Jul 28 20:43:22.053964 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:22.053927    2582 patch_prober.go:28] interesting pod/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Jul 28 20:43:22.054455 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:22.054025    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Jul 28 20:43:23.054124 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:23.054085    2582 patch_prober.go:28] interesting pod/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Jul 28 20:43:23.054578 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:23.054154    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Jul 28 20:43:24.054027 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:24.053988    2582 patch_prober.go:28] interesting pod/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Jul 28 20:43:24.054198 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:24.054043    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Jul 28 20:43:25.058398 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:25.058367    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:25.058813 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:25.058630    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:25.059316 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:25.059286    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:43:31.273819 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:31.273781    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:31.544684 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:31.544588    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:32.034308 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:32.034237    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:32.199166 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:32.199135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:34.654802 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:34.654763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:34.783918 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:34.783886    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"]
Jul 28 20:43:34.787886 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:34.787864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:34.790975 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:34.790955    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c0e826-serving-cert\""
Jul 28 20:43:34.832875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:34.832833    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"]
Jul 28 20:43:34.917164 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:34.917076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5714ff-3b8d-4672-b674-abed8a46a01c-maas-api-tls\") pod \"maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:34.917164 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:34.917157    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8pj\" (UniqueName: \"kubernetes.io/projected/2c5714ff-3b8d-4672-b674-abed8a46a01c-kube-api-access-8g8pj\") pod \"maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:34.917416 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:34.917187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5714ff-3b8d-4672-b674-abed8a46a01c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:34.931062 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:34.931007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:35.018490 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.018456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8pj\" (UniqueName: \"kubernetes.io/projected/2c5714ff-3b8d-4672-b674-abed8a46a01c-kube-api-access-8g8pj\") pod \"maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:35.018666 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.018499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5714ff-3b8d-4672-b674-abed8a46a01c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:35.018666 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.018564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5714ff-3b8d-4672-b674-abed8a46a01c-maas-api-tls\") pod \"maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:35.019276 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.019253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5714ff-3b8d-4672-b674-abed8a46a01c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:35.020957 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.020940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5714ff-3b8d-4672-b674-abed8a46a01c-maas-api-tls\") pod \"maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:35.030788 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.030752    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8pj\" (UniqueName: \"kubernetes.io/projected/2c5714ff-3b8d-4672-b674-abed8a46a01c-kube-api-access-8g8pj\") pod \"maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:35.098794 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.098760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:35.246363 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.246317    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:35.291471 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.291448    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"]
Jul 28 20:43:35.293633 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:43:35.293605    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5714ff_3b8d_4672_b674_abed8a46a01c.slice/crio-3cc044686082225d0f47b5456051af91ac6f57adf41ad5a909454463ecb248bd WatchSource:0}: Error finding container 3cc044686082225d0f47b5456051af91ac6f57adf41ad5a909454463ecb248bd: Status 404 returned error can't find the container with id 3cc044686082225d0f47b5456051af91ac6f57adf41ad5a909454463ecb248bd
Jul 28 20:43:35.811560 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.811519    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:35.966748 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:35.966718    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:36.269941 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:36.269896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc" event={"ID":"2c5714ff-3b8d-4672-b674-abed8a46a01c","Type":"ContainerStarted","Data":"eceeaecfc0e951460b89673775d787182eaa79aeca47ddb0d5f9c35edac47077"}
Jul 28 20:43:36.270134 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:36.269952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc" event={"ID":"2c5714ff-3b8d-4672-b674-abed8a46a01c","Type":"ContainerStarted","Data":"3cc044686082225d0f47b5456051af91ac6f57adf41ad5a909454463ecb248bd"}
Jul 28 20:43:36.270134 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:36.270005    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:36.301894 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:36.301842    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc" podStartSLOduration=1.763977814 podStartE2EDuration="2.301824952s" totalImagesPullingTime="537.847138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:43:36.300840889 +0000 UTC m=+2666.155151955" watchObservedRunningTime="2026-07-28 20:43:36.301824952 +0000 UTC m=+2666.156136017"
Jul 28 20:43:42.281144 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:42.281114    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:43:59.536571 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:43:59.536538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:03.394158 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:03.394083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:04.596513 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.596481    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"]
Jul 28 20:44:04.600605 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.600584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.604159 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.604137    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1b6922\"/\"default-dockercfg-qz22z\""
Jul 28 20:44:04.605223 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.605207    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1b6922\"/\"kube-root-ca.crt\""
Jul 28 20:44:04.605334 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.605317    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1b6922\"/\"openshift-service-ca.crt\""
Jul 28 20:44:04.605334 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.605326    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1b6922\"/\"auth-test-model-1b6922-kserve-self-signed-certs\""
Jul 28 20:44:04.612465 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.612428    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"]
Jul 28 20:44:04.641672 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.641632    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:04.691673 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.691638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-home\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.691673 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.691683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-model-cache\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.691892 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.691745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tls-certs\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.691892 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.691782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6jc\" (UniqueName: \"kubernetes.io/projected/351af7cb-0f21-4ee6-ad94-268595f4a2a4-kube-api-access-jc6jc\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.691892 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.691816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-dshm\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.691892 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.691832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tmp-dir\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.792694 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.792646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-home\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.792694 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.792707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-model-cache\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.792987 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.792748    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tls-certs\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.792987 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.792770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6jc\" (UniqueName: \"kubernetes.io/projected/351af7cb-0f21-4ee6-ad94-268595f4a2a4-kube-api-access-jc6jc\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.792987 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.792802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-dshm\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.792987 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.792820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tmp-dir\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.793210 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.793125    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-home\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.793210 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.793138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-model-cache\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.793310 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.793230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tmp-dir\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.795071 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.795047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-dshm\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.795365 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.795344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tls-certs\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.804629 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.804593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6jc\" (UniqueName: \"kubernetes.io/projected/351af7cb-0f21-4ee6-ad94-268595f4a2a4-kube-api-access-jc6jc\") pod \"auth-test-model-1b6922-kserve-585499c6c-4kbbm\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") " pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:04.910256 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:04.910164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:05.074564 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:05.074535    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"]
Jul 28 20:44:05.077107 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:44:05.077078    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod351af7cb_0f21_4ee6_ad94_268595f4a2a4.slice/crio-5585da78b77d5b41386a3b897ef446036ff31d4789f0ceeeabd51da47c579fe5 WatchSource:0}: Error finding container 5585da78b77d5b41386a3b897ef446036ff31d4789f0ceeeabd51da47c579fe5: Status 404 returned error can't find the container with id 5585da78b77d5b41386a3b897ef446036ff31d4789f0ceeeabd51da47c579fe5
Jul 28 20:44:05.422782 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:05.422739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm" event={"ID":"351af7cb-0f21-4ee6-ad94-268595f4a2a4","Type":"ContainerStarted","Data":"bbec7e593b15d03d233642c911bea2df28e2bb788a2b1c85254e5964c0c246e3"}
Jul 28 20:44:05.422782 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:05.422790    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm" event={"ID":"351af7cb-0f21-4ee6-ad94-268595f4a2a4","Type":"ContainerStarted","Data":"5585da78b77d5b41386a3b897ef446036ff31d4789f0ceeeabd51da47c579fe5"}
Jul 28 20:44:05.448877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:05.448818    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm" podStartSLOduration=1.448802389 podStartE2EDuration="1.448802389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:44:05.446807732 +0000 UTC m=+2695.301118820" watchObservedRunningTime="2026-07-28 20:44:05.448802389 +0000 UTC m=+2695.303113454"
Jul 28 20:44:06.293224 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:06.293163    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:10.950899 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:10.950863    2582 scope.go:117] "RemoveContainer" containerID="f1da379a8441593106d52ad7c326e4afb119648015f9350b9f30a27ff5884457"
Jul 28 20:44:14.910436 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:14.910406    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:14.910436 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:14.910451    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:14.923470 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:14.923439    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:15.484723 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:15.484693    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:44:49.678146 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:44:49.678101    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:00.149204 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:00.149166    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps"]
Jul 28 20:45:00.152769 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:00.152745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps"
Jul 28 20:45:00.157355 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:45:00.157323    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-cleanup-dockercfg-982zh\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" type="*v1.Secret"
Jul 28 20:45:00.157949 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:45:00.157925    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-29754525-mlvps\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="c3e303bb-1219-434e-9cc9-9ef44eb9d09c" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps"
Jul 28 20:45:00.172395 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:00.172364    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps"]
Jul 28 20:45:00.174432 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:00.174406    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx"]
Jul 28 20:45:00.186610 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:00.186580    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx"
Jul 28 20:45:00.202109 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:00.202077    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx"]
Jul 28 20:45:01.162769 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.162718    2582 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Jul 28 20:45:01.163176 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.162798    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps"
Jul 28 20:45:01.197358 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.197318    2582 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Jul 28 20:45:01.197528 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.197398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx"
Jul 28 20:45:01.339033 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.338998    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps"]
Jul 28 20:45:01.342884 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.342589    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\""
Jul 28 20:45:01.352815 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:45:01.352634    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e303bb_1219_434e_9cc9_9ef44eb9d09c.slice/crio-93ed394661da74ed7151e5025e03c06d59a0e5dd9dd703cfa5c7ebe8e19c5fdc WatchSource:0}: Error finding container 93ed394661da74ed7151e5025e03c06d59a0e5dd9dd703cfa5c7ebe8e19c5fdc: Status 404 returned error can't find the container with id 93ed394661da74ed7151e5025e03c06d59a0e5dd9dd703cfa5c7ebe8e19c5fdc
Jul 28 20:45:01.378285 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.378261    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx"]
Jul 28 20:45:01.379383 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:45:01.379361    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead2bfdb_2439_43a0_a589_08765357e4db.slice/crio-fa74dc1b7c92b35488878e37d4cd73ecf4e10f31351cba47e9c65241b9e26093 WatchSource:0}: Error finding container fa74dc1b7c92b35488878e37d4cd73ecf4e10f31351cba47e9c65241b9e26093: Status 404 returned error can't find the container with id fa74dc1b7c92b35488878e37d4cd73ecf4e10f31351cba47e9c65241b9e26093
Jul 28 20:45:01.705309 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.705255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps" event={"ID":"c3e303bb-1219-434e-9cc9-9ef44eb9d09c","Type":"ContainerStarted","Data":"24ca2e85d8a5cfb8305098f1b5717f9e88eeb8b1fbfc49f6af958e78a1d07d48"}
Jul 28 20:45:01.705505 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.705329    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps" event={"ID":"c3e303bb-1219-434e-9cc9-9ef44eb9d09c","Type":"ContainerStarted","Data":"93ed394661da74ed7151e5025e03c06d59a0e5dd9dd703cfa5c7ebe8e19c5fdc"}
Jul 28 20:45:01.706495 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.706471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx" event={"ID":"ead2bfdb-2439-43a0-a589-08765357e4db","Type":"ContainerStarted","Data":"fa74dc1b7c92b35488878e37d4cd73ecf4e10f31351cba47e9c65241b9e26093"}
Jul 28 20:45:01.726617 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:01.726565    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps" podStartSLOduration=1.726550898 podStartE2EDuration="1.726550898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:01.724542149 +0000 UTC m=+2751.578853242" watchObservedRunningTime="2026-07-28 20:45:01.726550898 +0000 UTC m=+2751.580861964"
Jul 28 20:45:02.712999 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:02.712905    2582 generic.go:356] "Generic (PLEG): container finished" podID="c3e303bb-1219-434e-9cc9-9ef44eb9d09c" containerID="24ca2e85d8a5cfb8305098f1b5717f9e88eeb8b1fbfc49f6af958e78a1d07d48" exitCode=0
Jul 28 20:45:02.713441 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:02.712991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps" event={"ID":"c3e303bb-1219-434e-9cc9-9ef44eb9d09c","Type":"ContainerDied","Data":"24ca2e85d8a5cfb8305098f1b5717f9e88eeb8b1fbfc49f6af958e78a1d07d48"}
Jul 28 20:45:02.714534 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:02.714508    2582 generic.go:356] "Generic (PLEG): container finished" podID="ead2bfdb-2439-43a0-a589-08765357e4db" containerID="924c4b13a6e68410db2cbbaee74d353d45fe9174b18667936b7a74fc2e9c79db" exitCode=0
Jul 28 20:45:02.714649 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:02.714580    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx" event={"ID":"ead2bfdb-2439-43a0-a589-08765357e4db","Type":"ContainerDied","Data":"924c4b13a6e68410db2cbbaee74d353d45fe9174b18667936b7a74fc2e9c79db"}
Jul 28 20:45:03.867620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:03.867593    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx"
Jul 28 20:45:03.899718 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:03.899589    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps"
Jul 28 20:45:04.726718 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:04.726685    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps"
Jul 28 20:45:04.726924 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:04.726684    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754525-mlvps" event={"ID":"c3e303bb-1219-434e-9cc9-9ef44eb9d09c","Type":"ContainerDied","Data":"93ed394661da74ed7151e5025e03c06d59a0e5dd9dd703cfa5c7ebe8e19c5fdc"}
Jul 28 20:45:04.726924 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:04.726819    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ed394661da74ed7151e5025e03c06d59a0e5dd9dd703cfa5c7ebe8e19c5fdc"
Jul 28 20:45:04.728122 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:04.728103    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx"
Jul 28 20:45:04.728266 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:04.728140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx" event={"ID":"ead2bfdb-2439-43a0-a589-08765357e4db","Type":"ContainerDied","Data":"fa74dc1b7c92b35488878e37d4cd73ecf4e10f31351cba47e9c65241b9e26093"}
Jul 28 20:45:04.728266 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:04.728188    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa74dc1b7c92b35488878e37d4cd73ecf4e10f31351cba47e9c65241b9e26093"
Jul 28 20:45:16.915740 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:16.915703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:16.983844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:45:16.983808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:10.984023 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:47:10.983983    2582 scope.go:117] "RemoveContainer" containerID="eb0cfc7c44b5a2de6d15f93b07f2c85eb77d09bd6a5efbce8af6ad96f403017e"
Jul 28 20:47:10.992695 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:47:10.992673    2582 scope.go:117] "RemoveContainer" containerID="7430a6ea9ff318f59cbfe5ee3a1f8b7436c2c821d27c9aebe6dc41644b5cfe85"
Jul 28 20:48:19.802464 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:48:19.802415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:21.352631 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:48:21.352573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:11.022969 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:11.022923    2582 scope.go:117] "RemoveContainer" containerID="1cc562d933c47754786447502471bbd72b8a73cc366f399ea4c67f80b1f620d2"
Jul 28 20:49:19.989862 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:19.989823    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"]
Jul 28 20:49:20.046553 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:20.046517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:20.916396 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:20.916320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:35.009012 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.008958    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm" podUID="351af7cb-0f21-4ee6-ad94-268595f4a2a4" containerName="main" containerID="cri-o://bbec7e593b15d03d233642c911bea2df28e2bb788a2b1c85254e5964c0c246e3" gracePeriod=45
Jul 28 20:49:35.148715 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.148684    2582 generic.go:356] "Generic (PLEG): container finished" podID="351af7cb-0f21-4ee6-ad94-268595f4a2a4" containerID="bbec7e593b15d03d233642c911bea2df28e2bb788a2b1c85254e5964c0c246e3" exitCode=0
Jul 28 20:49:35.148888 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.148752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm" event={"ID":"351af7cb-0f21-4ee6-ad94-268595f4a2a4","Type":"ContainerDied","Data":"bbec7e593b15d03d233642c911bea2df28e2bb788a2b1c85254e5964c0c246e3"}
Jul 28 20:49:35.266251 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.266186    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:49:35.340939 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.340903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-model-cache\") pod \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") "
Jul 28 20:49:35.341133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.340975    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc6jc\" (UniqueName: \"kubernetes.io/projected/351af7cb-0f21-4ee6-ad94-268595f4a2a4-kube-api-access-jc6jc\") pod \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") "
Jul 28 20:49:35.341133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.340996    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tls-certs\") pod \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") "
Jul 28 20:49:35.341133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.341030    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tmp-dir\") pod \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") "
Jul 28 20:49:35.341133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.341060    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-dshm\") pod \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") "
Jul 28 20:49:35.341133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.341106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-home\") pod \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\" (UID: \"351af7cb-0f21-4ee6-ad94-268595f4a2a4\") "
Jul 28 20:49:35.341431 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.341240    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "351af7cb-0f21-4ee6-ad94-268595f4a2a4" (UID: "351af7cb-0f21-4ee6-ad94-268595f4a2a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:49:35.341431 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.341336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "351af7cb-0f21-4ee6-ad94-268595f4a2a4" (UID: "351af7cb-0f21-4ee6-ad94-268595f4a2a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:49:35.341539 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.341470    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-home" (OuterVolumeSpecName: "home") pod "351af7cb-0f21-4ee6-ad94-268595f4a2a4" (UID: "351af7cb-0f21-4ee6-ad94-268595f4a2a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:49:35.343315 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.343278    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-dshm" (OuterVolumeSpecName: "dshm") pod "351af7cb-0f21-4ee6-ad94-268595f4a2a4" (UID: "351af7cb-0f21-4ee6-ad94-268595f4a2a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:49:35.343315 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.343286    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "351af7cb-0f21-4ee6-ad94-268595f4a2a4" (UID: "351af7cb-0f21-4ee6-ad94-268595f4a2a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:49:35.343436 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.343343    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351af7cb-0f21-4ee6-ad94-268595f4a2a4-kube-api-access-jc6jc" (OuterVolumeSpecName: "kube-api-access-jc6jc") pod "351af7cb-0f21-4ee6-ad94-268595f4a2a4" (UID: "351af7cb-0f21-4ee6-ad94-268595f4a2a4"). InnerVolumeSpecName "kube-api-access-jc6jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:35.442706 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.442671    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc6jc\" (UniqueName: \"kubernetes.io/projected/351af7cb-0f21-4ee6-ad94-268595f4a2a4-kube-api-access-jc6jc\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:49:35.442706 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.442702    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tls-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:49:35.442706 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.442712    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-tmp-dir\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:49:35.442952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.442720    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-dshm\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:49:35.442952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.442727    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-home\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:49:35.442952 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:35.442735    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351af7cb-0f21-4ee6-ad94-268595f4a2a4-model-cache\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:49:36.155203 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:36.155175    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"
Jul 28 20:49:36.155712 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:36.155211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm" event={"ID":"351af7cb-0f21-4ee6-ad94-268595f4a2a4","Type":"ContainerDied","Data":"5585da78b77d5b41386a3b897ef446036ff31d4789f0ceeeabd51da47c579fe5"}
Jul 28 20:49:36.155712 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:36.155266    2582 scope.go:117] "RemoveContainer" containerID="bbec7e593b15d03d233642c911bea2df28e2bb788a2b1c85254e5964c0c246e3"
Jul 28 20:49:36.178041 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:36.178012    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"]
Jul 28 20:49:36.187133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:36.187106    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/auth-test-model-1b6922-kserve-585499c6c-4kbbm"]
Jul 28 20:49:36.583453 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:36.583423    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351af7cb-0f21-4ee6-ad94-268595f4a2a4" path="/var/lib/kubelet/pods/351af7cb-0f21-4ee6-ad94-268595f4a2a4/volumes"
Jul 28 20:49:37.377755 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:37.377721    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:57.113050 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113012    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"]
Jul 28 20:49:57.113574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113450    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e303bb-1219-434e-9cc9-9ef44eb9d09c" containerName="cleanup"
Jul 28 20:49:57.113574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113463    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e303bb-1219-434e-9cc9-9ef44eb9d09c" containerName="cleanup"
Jul 28 20:49:57.113574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113480    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351af7cb-0f21-4ee6-ad94-268595f4a2a4" containerName="main"
Jul 28 20:49:57.113574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113486    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="351af7cb-0f21-4ee6-ad94-268595f4a2a4" containerName="main"
Jul 28 20:49:57.113574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113492    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead2bfdb-2439-43a0-a589-08765357e4db" containerName="cleanup"
Jul 28 20:49:57.113574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113498    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead2bfdb-2439-43a0-a589-08765357e4db" containerName="cleanup"
Jul 28 20:49:57.113574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113554    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="351af7cb-0f21-4ee6-ad94-268595f4a2a4" containerName="main"
Jul 28 20:49:57.113574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113562    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead2bfdb-2439-43a0-a589-08765357e4db" containerName="cleanup"
Jul 28 20:49:57.113574 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.113569    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e303bb-1219-434e-9cc9-9ef44eb9d09c" containerName="cleanup"
Jul 28 20:49:57.116564 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.116547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.120798 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.120776    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c0e826\"/\"kube-root-ca.crt\""
Jul 28 20:49:57.120927 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.120775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c0e826\"/\"default-dockercfg-w78mt\""
Jul 28 20:49:57.120927 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.120803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c0e826\"/\"sub-test-model-c0e826-kserve-self-signed-certs\""
Jul 28 20:49:57.120927 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.120806    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c0e826\"/\"openshift-service-ca.crt\""
Jul 28 20:49:57.130950 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.130924    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"]
Jul 28 20:49:57.169119 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.169084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:57.254426 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.254385    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-dshm\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.254426 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.254426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpd6\" (UniqueName: \"kubernetes.io/projected/69311425-3211-4d18-a4a0-4c6b1c6a4881-kube-api-access-zwpd6\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.254699 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.254463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-model-cache\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.254699 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.254484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69311425-3211-4d18-a4a0-4c6b1c6a4881-tls-certs\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.254699 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.254525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-tmp-dir\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.254699 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.254619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-home\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.355413 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.355372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-home\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.355590 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.355433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-dshm\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.355590 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.355461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpd6\" (UniqueName: \"kubernetes.io/projected/69311425-3211-4d18-a4a0-4c6b1c6a4881-kube-api-access-zwpd6\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.355590 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.355496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-model-cache\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.355590 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.355521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69311425-3211-4d18-a4a0-4c6b1c6a4881-tls-certs\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.355590 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.355565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-tmp-dir\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.355840 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.355810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-home\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.355914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.355866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-model-cache\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.356045 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.356024    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-tmp-dir\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.357934 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.357914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-dshm\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.358039 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.358004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69311425-3211-4d18-a4a0-4c6b1c6a4881-tls-certs\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.364927 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.364856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpd6\" (UniqueName: \"kubernetes.io/projected/69311425-3211-4d18-a4a0-4c6b1c6a4881-kube-api-access-zwpd6\") pod \"sub-test-model-c0e826-kserve-54455c5979-nscft\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") " pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.425684 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.425643    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:49:57.566336 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:57.566310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"]
Jul 28 20:49:57.568058 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:49:57.568033    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69311425_3211_4d18_a4a0_4c6b1c6a4881.slice/crio-4c799dbaea3c96991e56966242d0609dc0617aae71ddbd004e34d8294232fa6a WatchSource:0}: Error finding container 4c799dbaea3c96991e56966242d0609dc0617aae71ddbd004e34d8294232fa6a: Status 404 returned error can't find the container with id 4c799dbaea3c96991e56966242d0609dc0617aae71ddbd004e34d8294232fa6a
Jul 28 20:49:58.268335 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:58.268280    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft" event={"ID":"69311425-3211-4d18-a4a0-4c6b1c6a4881","Type":"ContainerStarted","Data":"4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15"}
Jul 28 20:49:58.268846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:58.268346    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft" event={"ID":"69311425-3211-4d18-a4a0-4c6b1c6a4881","Type":"ContainerStarted","Data":"4c799dbaea3c96991e56966242d0609dc0617aae71ddbd004e34d8294232fa6a"}
Jul 28 20:49:58.288531 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:49:58.288472    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft" podStartSLOduration=1.288457878 podStartE2EDuration="1.288457878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:58.286438013 +0000 UTC m=+3048.140749107" watchObservedRunningTime="2026-07-28 20:49:58.288457878 +0000 UTC m=+3048.142768943"
Jul 28 20:50:07.425797 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:50:07.425752    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:50:07.426260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:50:07.425914    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:50:07.440933 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:50:07.440907    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:50:08.330875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:50:08.330838    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:51:08.282706 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:08.282671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:09.008935 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:09.008890    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"]
Jul 28 20:51:09.035240 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:09.035209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:24.029678 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.029631    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft" podUID="69311425-3211-4d18-a4a0-4c6b1c6a4881" containerName="main" containerID="cri-o://4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15" gracePeriod=45
Jul 28 20:51:24.282657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.282602    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:51:24.354869 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.354832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-dshm\") pod \"69311425-3211-4d18-a4a0-4c6b1c6a4881\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") "
Jul 28 20:51:24.354869 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.354871    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-model-cache\") pod \"69311425-3211-4d18-a4a0-4c6b1c6a4881\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") "
Jul 28 20:51:24.355119 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.354895    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-tmp-dir\") pod \"69311425-3211-4d18-a4a0-4c6b1c6a4881\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") "
Jul 28 20:51:24.355119 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.354937    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69311425-3211-4d18-a4a0-4c6b1c6a4881-tls-certs\") pod \"69311425-3211-4d18-a4a0-4c6b1c6a4881\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") "
Jul 28 20:51:24.355119 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.354966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-home\") pod \"69311425-3211-4d18-a4a0-4c6b1c6a4881\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") "
Jul 28 20:51:24.355119 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.355006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwpd6\" (UniqueName: \"kubernetes.io/projected/69311425-3211-4d18-a4a0-4c6b1c6a4881-kube-api-access-zwpd6\") pod \"69311425-3211-4d18-a4a0-4c6b1c6a4881\" (UID: \"69311425-3211-4d18-a4a0-4c6b1c6a4881\") "
Jul 28 20:51:24.355354 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.355189    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-model-cache" (OuterVolumeSpecName: "model-cache") pod "69311425-3211-4d18-a4a0-4c6b1c6a4881" (UID: "69311425-3211-4d18-a4a0-4c6b1c6a4881"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:24.355354 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.355194    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69311425-3211-4d18-a4a0-4c6b1c6a4881" (UID: "69311425-3211-4d18-a4a0-4c6b1c6a4881"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:24.355354 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.355255    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-home" (OuterVolumeSpecName: "home") pod "69311425-3211-4d18-a4a0-4c6b1c6a4881" (UID: "69311425-3211-4d18-a4a0-4c6b1c6a4881"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:24.355473 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.355397    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-model-cache\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:51:24.355473 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.355423    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-tmp-dir\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:51:24.355473 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.355436    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-home\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:51:24.357173 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.357150    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-dshm" (OuterVolumeSpecName: "dshm") pod "69311425-3211-4d18-a4a0-4c6b1c6a4881" (UID: "69311425-3211-4d18-a4a0-4c6b1c6a4881"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:24.357253 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.357176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69311425-3211-4d18-a4a0-4c6b1c6a4881-kube-api-access-zwpd6" (OuterVolumeSpecName: "kube-api-access-zwpd6") pod "69311425-3211-4d18-a4a0-4c6b1c6a4881" (UID: "69311425-3211-4d18-a4a0-4c6b1c6a4881"). InnerVolumeSpecName "kube-api-access-zwpd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:51:24.357253 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.357155    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69311425-3211-4d18-a4a0-4c6b1c6a4881-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69311425-3211-4d18-a4a0-4c6b1c6a4881" (UID: "69311425-3211-4d18-a4a0-4c6b1c6a4881"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:51:24.456757 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.456714    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwpd6\" (UniqueName: \"kubernetes.io/projected/69311425-3211-4d18-a4a0-4c6b1c6a4881-kube-api-access-zwpd6\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:51:24.456757 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.456747    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69311425-3211-4d18-a4a0-4c6b1c6a4881-dshm\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:51:24.456757 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.456756    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69311425-3211-4d18-a4a0-4c6b1c6a4881-tls-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:51:24.710831 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.710800    2582 generic.go:356] "Generic (PLEG): container finished" podID="69311425-3211-4d18-a4a0-4c6b1c6a4881" containerID="4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15" exitCode=0
Jul 28 20:51:24.711020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.710841    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft" event={"ID":"69311425-3211-4d18-a4a0-4c6b1c6a4881","Type":"ContainerDied","Data":"4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15"}
Jul 28 20:51:24.711020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.710847    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"
Jul 28 20:51:24.711020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.710869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft" event={"ID":"69311425-3211-4d18-a4a0-4c6b1c6a4881","Type":"ContainerDied","Data":"4c799dbaea3c96991e56966242d0609dc0617aae71ddbd004e34d8294232fa6a"}
Jul 28 20:51:24.711020 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.710888    2582 scope.go:117] "RemoveContainer" containerID="4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15"
Jul 28 20:51:24.719894 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.719876    2582 scope.go:117] "RemoveContainer" containerID="4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15"
Jul 28 20:51:24.720180 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:51:24.720162    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15\": container with ID starting with 4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15 not found: ID does not exist" containerID="4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15"
Jul 28 20:51:24.720229 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.720191    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15"} err="failed to get container status \"4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15\": rpc error: code = NotFound desc = could not find container \"4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15\": container with ID starting with 4100a7d7b89665e06423e634e58bad20d1743902df85fedbc50164fda899ef15 not found: ID does not exist"
Jul 28 20:51:24.729415 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.729391    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"]
Jul 28 20:51:24.733455 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:24.733433    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c0e826/sub-test-model-c0e826-kserve-54455c5979-nscft"]
Jul 28 20:51:26.583614 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:26.583574    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69311425-3211-4d18-a4a0-4c6b1c6a4881" path="/var/lib/kubelet/pods/69311425-3211-4d18-a4a0-4c6b1c6a4881/volumes"
Jul 28 20:51:37.206410 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.206329    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"]
Jul 28 20:51:37.206792 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.206724    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69311425-3211-4d18-a4a0-4c6b1c6a4881" containerName="main"
Jul 28 20:51:37.206792 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.206737    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="69311425-3211-4d18-a4a0-4c6b1c6a4881" containerName="main"
Jul 28 20:51:37.206880 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.206812    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="69311425-3211-4d18-a4a0-4c6b1c6a4881" containerName="main"
Jul 28 20:51:37.210909 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.210885    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.215934 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.215912    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1b6922\"/\"rate-test-model-1b6922-kserve-self-signed-certs\""
Jul 28 20:51:37.216054 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.215978    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1b6922\"/\"openshift-service-ca.crt\""
Jul 28 20:51:37.216054 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.216017    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1b6922\"/\"kube-root-ca.crt\""
Jul 28 20:51:37.216148 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.215910    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1b6922\"/\"default-dockercfg-qz22z\""
Jul 28 20:51:37.223581 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.223555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"]
Jul 28 20:51:37.241241 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.241205    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:37.374303 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.374266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e27aa93-eed5-442e-a5af-deee8c640914-tls-certs\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.374482 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.374389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-tmp-dir\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.374482 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.374444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxmj\" (UniqueName: \"kubernetes.io/projected/7e27aa93-eed5-442e-a5af-deee8c640914-kube-api-access-5wxmj\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.374482 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.374479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-dshm\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.374592 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.374504    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-model-cache\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.374592 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.374558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-home\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.475494 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.475403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-dshm\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.475494 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.475443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-model-cache\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.475494 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.475473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-home\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.475773 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.475545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e27aa93-eed5-442e-a5af-deee8c640914-tls-certs\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.475773 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.475596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-tmp-dir\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.475773 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.475634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxmj\" (UniqueName: \"kubernetes.io/projected/7e27aa93-eed5-442e-a5af-deee8c640914-kube-api-access-5wxmj\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.476030 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.476003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-home\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.476152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.476054    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-model-cache\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.476152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.476124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-tmp-dir\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.477883 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.477863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-dshm\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.478253 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.478233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e27aa93-eed5-442e-a5af-deee8c640914-tls-certs\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.485148 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.485113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxmj\" (UniqueName: \"kubernetes.io/projected/7e27aa93-eed5-442e-a5af-deee8c640914-kube-api-access-5wxmj\") pod \"rate-test-model-1b6922-kserve-84694c787-zxmcl\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") " pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.521120 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.521081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:37.677278 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.677206    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"]
Jul 28 20:51:37.678878 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:51:37.678829    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e27aa93_eed5_442e_a5af_deee8c640914.slice/crio-aa644d5cd36e819e71601009814206dbe78904d7d8a9e930bb88cedbe8427a9f WatchSource:0}: Error finding container aa644d5cd36e819e71601009814206dbe78904d7d8a9e930bb88cedbe8427a9f: Status 404 returned error can't find the container with id aa644d5cd36e819e71601009814206dbe78904d7d8a9e930bb88cedbe8427a9f
Jul 28 20:51:37.778353 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.778266    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl" event={"ID":"7e27aa93-eed5-442e-a5af-deee8c640914","Type":"ContainerStarted","Data":"780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a"}
Jul 28 20:51:37.778353 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.778323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl" event={"ID":"7e27aa93-eed5-442e-a5af-deee8c640914","Type":"ContainerStarted","Data":"aa644d5cd36e819e71601009814206dbe78904d7d8a9e930bb88cedbe8427a9f"}
Jul 28 20:51:37.825398 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:37.825337    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl" podStartSLOduration=0.825320826 podStartE2EDuration="825.320826ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:37.823959502 +0000 UTC m=+3147.678270598" watchObservedRunningTime="2026-07-28 20:51:37.825320826 +0000 UTC m=+3147.679631891"
Jul 28 20:51:47.521623 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:47.521579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:47.522152 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:47.521644    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:47.534438 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:47.534413    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:47.847417 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:47.847331    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:51:55.321531 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:51:55.321491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:04.069674 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:52:04.069637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:52:45.621847 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:52:45.621814    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:05.777334 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:05.777282    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"]
Jul 28 20:53:05.800109 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:05.800081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:06.608561 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:06.608523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:20.795938 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:20.795892    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl" podUID="7e27aa93-eed5-442e-a5af-deee8c640914" containerName="main" containerID="cri-o://780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a" gracePeriod=45
Jul 28 20:53:21.063401 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.063374    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:53:21.143684 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.143651    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wxmj\" (UniqueName: \"kubernetes.io/projected/7e27aa93-eed5-442e-a5af-deee8c640914-kube-api-access-5wxmj\") pod \"7e27aa93-eed5-442e-a5af-deee8c640914\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") "
Jul 28 20:53:21.143873 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.143717    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-dshm\") pod \"7e27aa93-eed5-442e-a5af-deee8c640914\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") "
Jul 28 20:53:21.143873 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.143739    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-tmp-dir\") pod \"7e27aa93-eed5-442e-a5af-deee8c640914\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") "
Jul 28 20:53:21.143873 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.143761    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e27aa93-eed5-442e-a5af-deee8c640914-tls-certs\") pod \"7e27aa93-eed5-442e-a5af-deee8c640914\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") "
Jul 28 20:53:21.143873 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.143792    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-home\") pod \"7e27aa93-eed5-442e-a5af-deee8c640914\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") "
Jul 28 20:53:21.143873 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.143845    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-model-cache\") pod \"7e27aa93-eed5-442e-a5af-deee8c640914\" (UID: \"7e27aa93-eed5-442e-a5af-deee8c640914\") "
Jul 28 20:53:21.144135 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.144051    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e27aa93-eed5-442e-a5af-deee8c640914" (UID: "7e27aa93-eed5-442e-a5af-deee8c640914"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:21.144135 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.144124    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-tmp-dir\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:53:21.144135 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.144117    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-home" (OuterVolumeSpecName: "home") pod "7e27aa93-eed5-442e-a5af-deee8c640914" (UID: "7e27aa93-eed5-442e-a5af-deee8c640914"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:21.144256 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.144226    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e27aa93-eed5-442e-a5af-deee8c640914" (UID: "7e27aa93-eed5-442e-a5af-deee8c640914"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:21.145925 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.145904    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-dshm" (OuterVolumeSpecName: "dshm") pod "7e27aa93-eed5-442e-a5af-deee8c640914" (UID: "7e27aa93-eed5-442e-a5af-deee8c640914"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:21.146013 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.145992    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e27aa93-eed5-442e-a5af-deee8c640914-kube-api-access-5wxmj" (OuterVolumeSpecName: "kube-api-access-5wxmj") pod "7e27aa93-eed5-442e-a5af-deee8c640914" (UID: "7e27aa93-eed5-442e-a5af-deee8c640914"). InnerVolumeSpecName "kube-api-access-5wxmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:53:21.146055 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.146034    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e27aa93-eed5-442e-a5af-deee8c640914-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e27aa93-eed5-442e-a5af-deee8c640914" (UID: "7e27aa93-eed5-442e-a5af-deee8c640914"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:53:21.244875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.244837    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wxmj\" (UniqueName: \"kubernetes.io/projected/7e27aa93-eed5-442e-a5af-deee8c640914-kube-api-access-5wxmj\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:53:21.244875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.244869    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-dshm\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:53:21.244875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.244879    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e27aa93-eed5-442e-a5af-deee8c640914-tls-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:53:21.245119 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.244888    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-home\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:53:21.245119 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.244896    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e27aa93-eed5-442e-a5af-deee8c640914-model-cache\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:53:21.291109 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.291076    2582 generic.go:356] "Generic (PLEG): container finished" podID="7e27aa93-eed5-442e-a5af-deee8c640914" containerID="780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a" exitCode=0
Jul 28 20:53:21.291310 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.291121    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"
Jul 28 20:53:21.291310 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.291140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl" event={"ID":"7e27aa93-eed5-442e-a5af-deee8c640914","Type":"ContainerDied","Data":"780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a"}
Jul 28 20:53:21.291310 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.291171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl" event={"ID":"7e27aa93-eed5-442e-a5af-deee8c640914","Type":"ContainerDied","Data":"aa644d5cd36e819e71601009814206dbe78904d7d8a9e930bb88cedbe8427a9f"}
Jul 28 20:53:21.291310 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.291188    2582 scope.go:117] "RemoveContainer" containerID="780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a"
Jul 28 20:53:21.300696 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.300679    2582 scope.go:117] "RemoveContainer" containerID="780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a"
Jul 28 20:53:21.300955 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:53:21.300936    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a\": container with ID starting with 780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a not found: ID does not exist" containerID="780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a"
Jul 28 20:53:21.301021 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.300964    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a"} err="failed to get container status \"780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a\": rpc error: code = NotFound desc = could not find container \"780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a\": container with ID starting with 780bac47f7ee5c0c8d4aa461031dabc55c9c7ab5156e592e6b2fe1aa32e57c2a not found: ID does not exist"
Jul 28 20:53:21.313945 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.313917    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"]
Jul 28 20:53:21.317966 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:21.317944    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1b6922/rate-test-model-1b6922-kserve-84694c787-zxmcl"]
Jul 28 20:53:22.582871 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:22.582840    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e27aa93-eed5-442e-a5af-deee8c640914" path="/var/lib/kubelet/pods/7e27aa93-eed5-442e-a5af-deee8c640914/volumes"
Jul 28 20:53:22.884949 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:22.884867    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7507-139960181992448"]
Jul 28 20:53:22.885270 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:22.885257    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e27aa93-eed5-442e-a5af-deee8c640914" containerName="main"
Jul 28 20:53:22.885336 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:22.885271    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e27aa93-eed5-442e-a5af-deee8c640914" containerName="main"
Jul 28 20:53:22.885374 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:22.885363    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e27aa93-eed5-442e-a5af-deee8c640914" containerName="main"
Jul 28 20:53:22.888386 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:22.888369    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139960181992448"
Jul 28 20:53:22.891048 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:22.891026    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-26m6p\""
Jul 28 20:53:22.892885 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:22.892859    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139960181992448"]
Jul 28 20:53:22.957194 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:22.957165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dss\" (UniqueName: \"kubernetes.io/projected/156f7295-a6a3-4763-8c48-1093a9240ac2-kube-api-access-96dss\") pod \"test-curl-7507-139960181992448\" (UID: \"156f7295-a6a3-4763-8c48-1093a9240ac2\") " pod="opendatahub/test-curl-7507-139960181992448"
Jul 28 20:53:23.057742 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:23.057708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dss\" (UniqueName: \"kubernetes.io/projected/156f7295-a6a3-4763-8c48-1093a9240ac2-kube-api-access-96dss\") pod \"test-curl-7507-139960181992448\" (UID: \"156f7295-a6a3-4763-8c48-1093a9240ac2\") " pod="opendatahub/test-curl-7507-139960181992448"
Jul 28 20:53:23.067268 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:23.067232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dss\" (UniqueName: \"kubernetes.io/projected/156f7295-a6a3-4763-8c48-1093a9240ac2-kube-api-access-96dss\") pod \"test-curl-7507-139960181992448\" (UID: \"156f7295-a6a3-4763-8c48-1093a9240ac2\") " pod="opendatahub/test-curl-7507-139960181992448"
Jul 28 20:53:23.199177 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:23.199095    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139960181992448"
Jul 28 20:53:23.325128 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:23.325100    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139960181992448"]
Jul 28 20:53:23.327231 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:53:23.327201    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod156f7295_a6a3_4763_8c48_1093a9240ac2.slice/crio-a50c2fcf74035378920200472928508ebbabfa50a8da4abc751fdb7f87450045 WatchSource:0}: Error finding container a50c2fcf74035378920200472928508ebbabfa50a8da4abc751fdb7f87450045: Status 404 returned error can't find the container with id a50c2fcf74035378920200472928508ebbabfa50a8da4abc751fdb7f87450045
Jul 28 20:53:23.328588 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:23.328569    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:53:24.309473 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:24.309385    2582 generic.go:356] "Generic (PLEG): container finished" podID="156f7295-a6a3-4763-8c48-1093a9240ac2" containerID="2bedfd1ffa99b3cc8c03f3def4d0a49fe8d8c77de9930acbd0354c47e6588af6" exitCode=0
Jul 28 20:53:24.309834 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:24.309469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139960181992448" event={"ID":"156f7295-a6a3-4763-8c48-1093a9240ac2","Type":"ContainerDied","Data":"2bedfd1ffa99b3cc8c03f3def4d0a49fe8d8c77de9930acbd0354c47e6588af6"}
Jul 28 20:53:24.309834 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:24.309510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139960181992448" event={"ID":"156f7295-a6a3-4763-8c48-1093a9240ac2","Type":"ContainerStarted","Data":"a50c2fcf74035378920200472928508ebbabfa50a8da4abc751fdb7f87450045"}
Jul 28 20:53:25.447599 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:25.447572    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139960181992448"
Jul 28 20:53:25.470685 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:25.470663    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7507-139960181992448_156f7295-a6a3-4763-8c48-1093a9240ac2/test-curl-7507-139960181992448/0.log"
Jul 28 20:53:25.475900 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:25.475881    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96dss\" (UniqueName: \"kubernetes.io/projected/156f7295-a6a3-4763-8c48-1093a9240ac2-kube-api-access-96dss\") pod \"156f7295-a6a3-4763-8c48-1093a9240ac2\" (UID: \"156f7295-a6a3-4763-8c48-1093a9240ac2\") "
Jul 28 20:53:25.477872 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:25.477848    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156f7295-a6a3-4763-8c48-1093a9240ac2-kube-api-access-96dss" (OuterVolumeSpecName: "kube-api-access-96dss") pod "156f7295-a6a3-4763-8c48-1093a9240ac2" (UID: "156f7295-a6a3-4763-8c48-1093a9240ac2"). InnerVolumeSpecName "kube-api-access-96dss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:53:25.495107 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:25.495079    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7507-139960181992448"]
Jul 28 20:53:25.498846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:25.498828    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7507-139960181992448"]
Jul 28 20:53:25.576603 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:25.576528    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96dss\" (UniqueName: \"kubernetes.io/projected/156f7295-a6a3-4763-8c48-1093a9240ac2-kube-api-access-96dss\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:53:26.047018 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.046985    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:26.323313 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.321982    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139960181992448"
Jul 28 20:53:26.323313 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.322022    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a50c2fcf74035378920200472928508ebbabfa50a8da4abc751fdb7f87450045"
Jul 28 20:53:26.330934 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.330901    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"]
Jul 28 20:53:26.331514 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.331489    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156f7295-a6a3-4763-8c48-1093a9240ac2" containerName="test-curl-7507-139960181992448"
Jul 28 20:53:26.331514 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.331513    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="156f7295-a6a3-4763-8c48-1093a9240ac2" containerName="test-curl-7507-139960181992448"
Jul 28 20:53:26.331722 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.331620    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="156f7295-a6a3-4763-8c48-1093a9240ac2" containerName="test-curl-7507-139960181992448"
Jul 28 20:53:26.335026 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.335005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.336463 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:53:26.336436    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7507-139960181992448\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="156f7295-a6a3-4763-8c48-1093a9240ac2" pod="opendatahub/test-curl-7507-139960181992448"
Jul 28 20:53:26.337556 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.337540    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2dfc5b-openshift-default-dockercfg-tccbq\""
Jul 28 20:53:26.349696 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.349676    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"]
Jul 28 20:53:26.371686 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:53:26.371657    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7507-139960181992448\" is forbidden: User \"system:node:ip-10-0-128-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-43.ec2.internal' and this object" podUID="156f7295-a6a3-4763-8c48-1093a9240ac2" pod="opendatahub/test-curl-7507-139960181992448"
Jul 28 20:53:26.381709 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.381688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-certs\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.381832 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.381728    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-podinfo\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.381832 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.381792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istiod-ca-cert\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.381950 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.381834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-data\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.381950 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.381868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-envoy\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.382034 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.381969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-credential-socket\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.382034 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.381995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-token\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.382034 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.382019    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-socket\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.382163 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.382034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nztz9\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-kube-api-access-nztz9\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.482728 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.482697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-credential-socket\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.482728 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.482732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-token\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.482749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-socket\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.482868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nztz9\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-kube-api-access-nztz9\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.482938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-certs\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.482981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-podinfo\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.483037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istiod-ca-cert\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.483090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-data\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483260 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.483125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-envoy\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.483284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-socket\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.483376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-credential-socket\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.483425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-certs\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483647 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.483489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-data\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.483917 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.483894    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istiod-ca-cert\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.485433 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.485409    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-envoy\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.485618 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.485601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-podinfo\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.491520 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.491493    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nztz9\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-kube-api-access-nztz9\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.491616 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.491524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-token\") pod \"e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") " pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.584470 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.584389    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156f7295-a6a3-4763-8c48-1093a9240ac2" path="/var/lib/kubelet/pods/156f7295-a6a3-4763-8c48-1093a9240ac2/volumes"
Jul 28 20:53:26.646391 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.646364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:26.793107 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.793083    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"]
Jul 28 20:53:26.795155 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:53:26.795127    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a41f99b_0d0e_4a5c_8e37_7bcfc0576d1f.slice/crio-43440281c4f5e6c877342b11b764b9f97c23a4f663b673b32428f13e08ddba3c WatchSource:0}: Error finding container 43440281c4f5e6c877342b11b764b9f97c23a4f663b673b32428f13e08ddba3c: Status 404 returned error can't find the container with id 43440281c4f5e6c877342b11b764b9f97c23a4f663b673b32428f13e08ddba3c
Jul 28 20:53:26.797589 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.797545    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:53:26.797668 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.797643    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:53:26.797705 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:26.797691    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:53:27.328734 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:27.328699    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" event={"ID":"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f","Type":"ContainerStarted","Data":"fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb"}
Jul 28 20:53:27.328893 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:27.328748    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" event={"ID":"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f","Type":"ContainerStarted","Data":"43440281c4f5e6c877342b11b764b9f97c23a4f663b673b32428f13e08ddba3c"}
Jul 28 20:53:27.351445 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:27.351393    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" podStartSLOduration=1.351378988 podStartE2EDuration="1.351378988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:27.349404078 +0000 UTC m=+3257.203715147" watchObservedRunningTime="2026-07-28 20:53:27.351378988 +0000 UTC m=+3257.205690054"
Jul 28 20:53:27.647170 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:27.647078    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:27.652274 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:27.652249    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:28.334368 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:28.334337    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:28.335302 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:28.335274    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:28.359856 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:28.359827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:29.382544 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.382510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:29.478894 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.478859    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:29.480400 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.480368    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"]
Jul 28 20:53:29.484335 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.484315    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:29.487612 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.487591    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2dfc5b-serving-cert\""
Jul 28 20:53:29.498405 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.498381    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"]
Jul 28 20:53:29.512672 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.512644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146d4f9d-b4ab-4b9e-a417-079afc86f224-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:29.512793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.512688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvsv6\" (UniqueName: \"kubernetes.io/projected/146d4f9d-b4ab-4b9e-a417-079afc86f224-kube-api-access-hvsv6\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:29.512846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.512829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146d4f9d-b4ab-4b9e-a417-079afc86f224-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:29.566999 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.566968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:29.613649 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.613619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146d4f9d-b4ab-4b9e-a417-079afc86f224-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:29.613826 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.613715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146d4f9d-b4ab-4b9e-a417-079afc86f224-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:29.613826 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.613738    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvsv6\" (UniqueName: \"kubernetes.io/projected/146d4f9d-b4ab-4b9e-a417-079afc86f224-kube-api-access-hvsv6\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:29.613826 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:53:29.613804    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-serving-cert: secret "maas-api-e2e-ipp-a-2dfc5b-serving-cert" not found
Jul 28 20:53:29.614032 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:53:29.613899    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146d4f9d-b4ab-4b9e-a417-079afc86f224-maas-api-tls podName:146d4f9d-b4ab-4b9e-a417-079afc86f224 nodeName:}" failed. No retries permitted until 2026-07-28 20:53:30.113876902 +0000 UTC m=+3259.968187963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/146d4f9d-b4ab-4b9e-a417-079afc86f224-maas-api-tls") pod "maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp" (UID: "146d4f9d-b4ab-4b9e-a417-079afc86f224") : secret "maas-api-e2e-ipp-a-2dfc5b-serving-cert" not found
Jul 28 20:53:29.614413 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.614393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146d4f9d-b4ab-4b9e-a417-079afc86f224-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:29.623862 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.623835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvsv6\" (UniqueName: \"kubernetes.io/projected/146d4f9d-b4ab-4b9e-a417-079afc86f224-kube-api-access-hvsv6\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:29.846355 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:29.846318    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:30.111472 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:30.111392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:30.118635 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:30.118614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146d4f9d-b4ab-4b9e-a417-079afc86f224-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:30.120965 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:30.120945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146d4f9d-b4ab-4b9e-a417-079afc86f224-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:30.207603 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:30.207575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:30.396885 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:30.396793    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:30.541865 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:30.541835    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"]
Jul 28 20:53:30.543554 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:53:30.543519    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146d4f9d_b4ab_4b9e_a417_079afc86f224.slice/crio-7c8edd3b4e1cc0e04140bc45aca053d9c1c111627eb1ac69194db973f4164c5b WatchSource:0}: Error finding container 7c8edd3b4e1cc0e04140bc45aca053d9c1c111627eb1ac69194db973f4164c5b: Status 404 returned error can't find the container with id 7c8edd3b4e1cc0e04140bc45aca053d9c1c111627eb1ac69194db973f4164c5b
Jul 28 20:53:31.351598 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:31.351567    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp" event={"ID":"146d4f9d-b4ab-4b9e-a417-079afc86f224","Type":"ContainerStarted","Data":"a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e"}
Jul 28 20:53:31.351774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:31.351623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp" event={"ID":"146d4f9d-b4ab-4b9e-a417-079afc86f224","Type":"ContainerStarted","Data":"7c8edd3b4e1cc0e04140bc45aca053d9c1c111627eb1ac69194db973f4164c5b"}
Jul 28 20:53:31.351774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:31.351668    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:31.370462 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:31.370410    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp" podStartSLOduration=1.7113658520000001 podStartE2EDuration="2.37039154s" totalImagesPullingTime="659.025688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:31.368645542 +0000 UTC m=+3261.222956624" watchObservedRunningTime="2026-07-28 20:53:31.37039154 +0000 UTC m=+3261.224702607"
Jul 28 20:53:37.362139 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:37.362111    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:53:38.971282 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:38.971236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:44.723691 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:44.723645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:45.002911 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.002874    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"]
Jul 28 20:53:45.009285 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.009250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.012083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.012037    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1ac855-openshift-default-dockercfg-hmm9k\""
Jul 28 20:53:45.018691 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.018663    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"]
Jul 28 20:53:45.033710 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.033680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:45.080150 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.080102    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-socket\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.080341 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.080170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-credential-socket\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.080341 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.080223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-istio-token\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.080341 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.080266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-envoy\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.080477 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.080354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxcp\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-kube-api-access-qdxcp\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.080477 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.080403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b478645-fe9a-4429-910f-cbfff26a754c-istio-podinfo\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.080477 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.080442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-data\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.080593 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.080514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-certs\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.080637 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.080604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b478645-fe9a-4429-910f-cbfff26a754c-istiod-ca-cert\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.181615 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.181580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b478645-fe9a-4429-910f-cbfff26a754c-istio-podinfo\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.181815 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.181623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-data\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.181815 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.181654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-certs\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.181815 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.181766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b478645-fe9a-4429-910f-cbfff26a754c-istiod-ca-cert\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.181815 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.181803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-socket\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.182046 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.181827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-credential-socket\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.182046 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.181873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-istio-token\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.182046 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.181909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-envoy\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.182046 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.181930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxcp\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-kube-api-access-qdxcp\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.182712 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.182430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-credential-socket\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.182712 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.182636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-socket\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.182712 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.182661    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b478645-fe9a-4429-910f-cbfff26a754c-istiod-ca-cert\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.182984 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.182938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-certs\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.182984 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.182966    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-data\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.184620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.184595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b478645-fe9a-4429-910f-cbfff26a754c-istio-podinfo\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.184730 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.184620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-envoy\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.190758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.190736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxcp\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-kube-api-access-qdxcp\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.191265 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.191244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-istio-token\") pod \"e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") " pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.203816 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.203792    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:45.337411 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.337311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:45.491238 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.491206    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"]
Jul 28 20:53:45.494083 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:53:45.494055    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b478645_fe9a_4429_910f_cbfff26a754c.slice/crio-d2ae1c54eed1209c8560baba2ae92500467d3d492616e1cac5d72eccab75b814 WatchSource:0}: Error finding container d2ae1c54eed1209c8560baba2ae92500467d3d492616e1cac5d72eccab75b814: Status 404 returned error can't find the container with id d2ae1c54eed1209c8560baba2ae92500467d3d492616e1cac5d72eccab75b814
Jul 28 20:53:45.496962 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.496923    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:53:45.497057 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.497022    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:53:45.497098 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:45.497071    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:53:46.432605 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:46.432573    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerID="fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb" exitCode=0
Jul 28 20:53:46.433148 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:46.432641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" event={"ID":"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f","Type":"ContainerDied","Data":"fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb"}
Jul 28 20:53:46.433148 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:46.433118    2582 scope.go:117] "RemoveContainer" containerID="fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb"
Jul 28 20:53:46.434285 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:46.434252    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:53:46.434415 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:46.434355    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:53:46.434415 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:46.434394    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:53:46.434591 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:46.434569    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" event={"ID":"3b478645-fe9a-4429-910f-cbfff26a754c","Type":"ContainerStarted","Data":"5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7"}
Jul 28 20:53:46.434632 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:46.434623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" event={"ID":"3b478645-fe9a-4429-910f-cbfff26a754c","Type":"ContainerStarted","Data":"d2ae1c54eed1209c8560baba2ae92500467d3d492616e1cac5d72eccab75b814"}
Jul 28 20:53:46.474859 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:46.474767    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" podStartSLOduration=2.474750363 podStartE2EDuration="2.474750363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:46.473209497 +0000 UTC m=+3276.327520576" watchObservedRunningTime="2026-07-28 20:53:46.474750363 +0000 UTC m=+3276.329061429"
Jul 28 20:53:47.337568 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:47.337531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:47.343038 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:47.343013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:47.451231 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:47.451187    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" event={"ID":"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f","Type":"ContainerStarted","Data":"8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f"}
Jul 28 20:53:47.451741 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:47.451424    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:47.452440 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:47.452420    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:53:47.647466 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:47.647386    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:47.647720 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:47.647697    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Jul 28 20:53:47.647773 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:47.647747    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Jul 28 20:53:48.386324 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:48.386255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:48.503032 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:48.502998    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:48.647935 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:48.647814    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Jul 28 20:53:48.647935 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:48.647899    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Jul 28 20:53:49.647097 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:49.647053    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Jul 28 20:53:49.647650 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:49.647129    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Jul 28 20:53:50.295806 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:50.295761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:50.647655 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:50.647551    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Jul 28 20:53:50.647655 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:50.647634    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Jul 28 20:53:50.712521 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:50.712479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:50.815801 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:50.815759    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:50.913829 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:50.913748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:53:51.646821 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:51.646780    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Jul 28 20:53:51.647192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:51.647149    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Jul 28 20:53:52.650896 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:52.650864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:52.651280 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:52.651233    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:53:52.651883 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:53:52.651864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:54:03.493469 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:03.493426    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 20:54:03.493887 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:03.493494    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 20:54:04.162857 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:04.162821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:05.553424 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:05.553394    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b478645-fe9a-4429-910f-cbfff26a754c" containerID="5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7" exitCode=0
Jul 28 20:54:05.553875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:05.553430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" event={"ID":"3b478645-fe9a-4429-910f-cbfff26a754c","Type":"ContainerDied","Data":"5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7"}
Jul 28 20:54:05.553875 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:05.553852    2582 scope.go:117] "RemoveContainer" containerID="5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7"
Jul 28 20:54:05.554936 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:05.554906    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:54:05.554991 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:05.554978    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:54:05.555036 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:05.555015    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 20:54:06.560716 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:06.560674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" event={"ID":"3b478645-fe9a-4429-910f-cbfff26a754c","Type":"ContainerStarted","Data":"8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf"}
Jul 28 20:54:07.337783 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:07.337690    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:54:07.338004 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:07.337943    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Jul 28 20:54:07.338084 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:07.338001    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Jul 28 20:54:08.338550 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:08.338513    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Jul 28 20:54:08.339186 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:08.339154    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Jul 28 20:54:09.338256 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:09.338219    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Jul 28 20:54:09.338679 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:09.338650    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Jul 28 20:54:09.992486 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:09.992449    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5gbgl_988d2ddd-e609-4125-8a53-b6904e158e7a/payload-processing/0.log"
Jul 28 20:54:10.338310 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:10.338196    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Jul 28 20:54:10.338567 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:10.338539    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Jul 28 20:54:11.341737 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:11.341651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:54:11.342190 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:11.342069    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:54:11.342947 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:11.342926    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:54:12.459307 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:12.459262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:12.559343 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:12.559303    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:27.711339 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:27.711305    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:48.712350 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:48.712257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:54:48.811743 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:54:48.811689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:12.555150 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:12.555119    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5gbgl_988d2ddd-e609-4125-8a53-b6904e158e7a/payload-processing/0.log"
Jul 28 20:55:12.947255 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:12.947166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:13.689896 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:13.689860    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:14.218066 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:14.218035    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:15.422771 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:15.422735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:16.570085 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:16.570041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:16.671182 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:16.671148    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:18.743233 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:18.743188    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:19.234725 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:19.234696    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:19.376441 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:19.376398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:19.474540 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:19.474491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:31.354343 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:31.354286    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:49.206844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:49.206808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:49.319313 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:49.319272    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:49.451114 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:49.451077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:49.554567 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:49.554534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:50.048096 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:50.048063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:56.561356 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:56.561314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:56.664387 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:56.664348    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:56.871543 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:56.871461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:57.112221 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:55:57.112190    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:00.693865 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:00.693831    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"]
Jul 28 20:56:00.697388 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:00.697371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"
Jul 28 20:56:00.699775 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:00.699754    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\""
Jul 28 20:56:00.706270 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:00.706243    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"]
Jul 28 20:56:00.732430 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:00.732393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f3ccc0-a439-415a-a38a-4f98b13a38a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj\" (UID: \"00f3ccc0-a439-415a-a38a-4f98b13a38a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"
Jul 28 20:56:00.833795 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:00.833758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f3ccc0-a439-415a-a38a-4f98b13a38a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj\" (UID: \"00f3ccc0-a439-415a-a38a-4f98b13a38a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"
Jul 28 20:56:00.834418 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:00.834399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f3ccc0-a439-415a-a38a-4f98b13a38a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj\" (UID: \"00f3ccc0-a439-415a-a38a-4f98b13a38a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"
Jul 28 20:56:01.008725 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:01.008685    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"
Jul 28 20:56:01.145791 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:01.145756    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"]
Jul 28 20:56:01.153490 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:56:01.153458    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f3ccc0_a439_415a_a38a_4f98b13a38a8.slice/crio-973f06027f5cdcd23f342d92328a44637e1ab40baf9b7436219a268074de168c WatchSource:0}: Error finding container 973f06027f5cdcd23f342d92328a44637e1ab40baf9b7436219a268074de168c: Status 404 returned error can't find the container with id 973f06027f5cdcd23f342d92328a44637e1ab40baf9b7436219a268074de168c
Jul 28 20:56:01.185596 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:01.185562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj" event={"ID":"00f3ccc0-a439-415a-a38a-4f98b13a38a8","Type":"ContainerStarted","Data":"973f06027f5cdcd23f342d92328a44637e1ab40baf9b7436219a268074de168c"}
Jul 28 20:56:02.192427 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:02.192395    2582 generic.go:356] "Generic (PLEG): container finished" podID="00f3ccc0-a439-415a-a38a-4f98b13a38a8" containerID="e9a5e8b796ddb243ae5fcf782f28e696c14dc80e83c569dbcda27eea02455615" exitCode=0
Jul 28 20:56:02.192827 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:02.192485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj" event={"ID":"00f3ccc0-a439-415a-a38a-4f98b13a38a8","Type":"ContainerDied","Data":"e9a5e8b796ddb243ae5fcf782f28e696c14dc80e83c569dbcda27eea02455615"}
Jul 28 20:56:03.339511 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:03.339487    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"
Jul 28 20:56:03.461740 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:03.461645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f3ccc0-a439-415a-a38a-4f98b13a38a8-maas-api-service-ca\") pod \"00f3ccc0-a439-415a-a38a-4f98b13a38a8\" (UID: \"00f3ccc0-a439-415a-a38a-4f98b13a38a8\") "
Jul 28 20:56:03.462050 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:03.462026    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f3ccc0-a439-415a-a38a-4f98b13a38a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00f3ccc0-a439-415a-a38a-4f98b13a38a8" (UID: "00f3ccc0-a439-415a-a38a-4f98b13a38a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:56:03.563022 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:03.562984    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f3ccc0-a439-415a-a38a-4f98b13a38a8-maas-api-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:56:04.204839 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:04.204812    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"
Jul 28 20:56:04.204839 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:04.204832    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj" event={"ID":"00f3ccc0-a439-415a-a38a-4f98b13a38a8","Type":"ContainerDied","Data":"973f06027f5cdcd23f342d92328a44637e1ab40baf9b7436219a268074de168c"}
Jul 28 20:56:04.205080 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:04.204875    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="973f06027f5cdcd23f342d92328a44637e1ab40baf9b7436219a268074de168c"
Jul 28 20:56:19.205246 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.205093    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"]
Jul 28 20:56:19.205852 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.205452    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp" podUID="146d4f9d-b4ab-4b9e-a417-079afc86f224" containerName="maas-api" containerID="cri-o://a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e" gracePeriod=30
Jul 28 20:56:19.243029 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.242994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:19.343580 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.343539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:19.443233 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.443174    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:19.491810 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.491781    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:56:19.517860 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.517823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:19.577986 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.577950    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"]
Jul 28 20:56:19.584640 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.584615    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dfc5b-ptdqj"]
Jul 28 20:56:19.630810 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.630784    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvsv6\" (UniqueName: \"kubernetes.io/projected/146d4f9d-b4ab-4b9e-a417-079afc86f224-kube-api-access-hvsv6\") pod \"146d4f9d-b4ab-4b9e-a417-079afc86f224\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") "
Jul 28 20:56:19.630975 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.630838    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146d4f9d-b4ab-4b9e-a417-079afc86f224-maas-api-tls\") pod \"146d4f9d-b4ab-4b9e-a417-079afc86f224\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") "
Jul 28 20:56:19.630975 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.630871    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146d4f9d-b4ab-4b9e-a417-079afc86f224-openshift-service-ca\") pod \"146d4f9d-b4ab-4b9e-a417-079afc86f224\" (UID: \"146d4f9d-b4ab-4b9e-a417-079afc86f224\") "
Jul 28 20:56:19.631375 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.631346    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146d4f9d-b4ab-4b9e-a417-079afc86f224-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "146d4f9d-b4ab-4b9e-a417-079afc86f224" (UID: "146d4f9d-b4ab-4b9e-a417-079afc86f224"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:56:19.633013 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.632985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146d4f9d-b4ab-4b9e-a417-079afc86f224-kube-api-access-hvsv6" (OuterVolumeSpecName: "kube-api-access-hvsv6") pod "146d4f9d-b4ab-4b9e-a417-079afc86f224" (UID: "146d4f9d-b4ab-4b9e-a417-079afc86f224"). InnerVolumeSpecName "kube-api-access-hvsv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:56:19.633127 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.633018    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146d4f9d-b4ab-4b9e-a417-079afc86f224-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "146d4f9d-b4ab-4b9e-a417-079afc86f224" (UID: "146d4f9d-b4ab-4b9e-a417-079afc86f224"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:56:19.732567 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.732532    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvsv6\" (UniqueName: \"kubernetes.io/projected/146d4f9d-b4ab-4b9e-a417-079afc86f224-kube-api-access-hvsv6\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:56:19.732733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.732571    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146d4f9d-b4ab-4b9e-a417-079afc86f224-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:56:19.732733 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:19.732586    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146d4f9d-b4ab-4b9e-a417-079afc86f224-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:56:20.044386 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.044271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:20.292814 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.292779    2582 generic.go:356] "Generic (PLEG): container finished" podID="146d4f9d-b4ab-4b9e-a417-079afc86f224" containerID="a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e" exitCode=0
Jul 28 20:56:20.293278 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.292834    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"
Jul 28 20:56:20.293278 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.292833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp" event={"ID":"146d4f9d-b4ab-4b9e-a417-079afc86f224","Type":"ContainerDied","Data":"a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e"}
Jul 28 20:56:20.293278 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.292972    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp" event={"ID":"146d4f9d-b4ab-4b9e-a417-079afc86f224","Type":"ContainerDied","Data":"7c8edd3b4e1cc0e04140bc45aca053d9c1c111627eb1ac69194db973f4164c5b"}
Jul 28 20:56:20.293278 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.293006    2582 scope.go:117] "RemoveContainer" containerID="a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e"
Jul 28 20:56:20.302986 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.302967    2582 scope.go:117] "RemoveContainer" containerID="a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e"
Jul 28 20:56:20.303242 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:56:20.303224    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e\": container with ID starting with a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e not found: ID does not exist" containerID="a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e"
Jul 28 20:56:20.303313 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.303252    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e"} err="failed to get container status \"a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e\": rpc error: code = NotFound desc = could not find container \"a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e\": container with ID starting with a32be76f8662fa8bbaa5d4693b9cac3684bb983acfc94b45cc81163b7cb61f1e not found: ID does not exist"
Jul 28 20:56:20.315903 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.315879    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"]
Jul 28 20:56:20.321482 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.321461    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2dfc5b-75bfd8d7db-xj2pp"]
Jul 28 20:56:20.582961 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.582885    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f3ccc0-a439-415a-a38a-4f98b13a38a8" path="/var/lib/kubelet/pods/00f3ccc0-a439-415a-a38a-4f98b13a38a8/volumes"
Jul 28 20:56:20.583254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:20.583242    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146d4f9d-b4ab-4b9e-a417-079afc86f224" path="/var/lib/kubelet/pods/146d4f9d-b4ab-4b9e-a417-079afc86f224/volumes"
Jul 28 20:56:25.037404 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:25.037366    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:58.464725 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:56:58.464689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:01.813497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:01.813465    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:02.527559 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:02.527514    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:08.767786 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:08.767746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:08.866972 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:08.866925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:09.034563 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:09.034469    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"]
Jul 28 20:57:09.034792 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:09.034765    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy" containerID="cri-o://8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f" gracePeriod=30
Jul 28 20:57:09.050732 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:09.050693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:09.293443 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:09.293337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:14.299022 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.298958    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:57:14.371485 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371448    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-credential-socket\") pod \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") "
Jul 28 20:57:14.371681 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371507    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-token\") pod \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") "
Jul 28 20:57:14.371681 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-data\") pod \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") "
Jul 28 20:57:14.371681 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371557    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-socket\") pod \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") "
Jul 28 20:57:14.371681 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371590    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istiod-ca-cert\") pod \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") "
Jul 28 20:57:14.371681 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371623    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nztz9\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-kube-api-access-nztz9\") pod \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") "
Jul 28 20:57:14.371681 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371652    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-podinfo\") pod \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") "
Jul 28 20:57:14.371986 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371698    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-certs\") pod \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") "
Jul 28 20:57:14.371986 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371785    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-envoy\") pod \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\" (UID: \"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f\") "
Jul 28 20:57:14.372086 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371778    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" (UID: "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:14.372141 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.371926    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" (UID: "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:14.372141 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.372005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" (UID: "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:14.372369 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.372317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" (UID: "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:57:14.372369 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.372350    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" (UID: "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:14.373912 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.373878    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" (UID: "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:14.373912 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.373880    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" (UID: "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:57:14.374081 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.373969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-kube-api-access-nztz9" (OuterVolumeSpecName: "kube-api-access-nztz9") pod "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" (UID: "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f"). InnerVolumeSpecName "kube-api-access-nztz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:57:14.374081 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.374010    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" (UID: "7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:57:14.473497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.473458    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-envoy\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:14.473497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.473484    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-credential-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:14.473497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.473494    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:14.473497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.473503    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-data\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:14.473497 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.473510    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:14.473889 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.473519    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istiod-ca-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:14.473889 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.473528    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nztz9\" (UniqueName: \"kubernetes.io/projected/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-kube-api-access-nztz9\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:14.473889 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.473535    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-istio-podinfo\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:14.473889 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.473543    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f-workload-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:14.577661 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.577573    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerID="8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f" exitCode=0
Jul 28 20:57:14.577806 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.577679    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"
Jul 28 20:57:14.582532 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.582507    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" event={"ID":"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f","Type":"ContainerDied","Data":"8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f"}
Jul 28 20:57:14.582678 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.582546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb" event={"ID":"7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f","Type":"ContainerDied","Data":"43440281c4f5e6c877342b11b764b9f97c23a4f663b673b32428f13e08ddba3c"}
Jul 28 20:57:14.582678 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.582568    2582 scope.go:117] "RemoveContainer" containerID="8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f"
Jul 28 20:57:14.592679 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.592662    2582 scope.go:117] "RemoveContainer" containerID="fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb"
Jul 28 20:57:14.600357 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.600339    2582 scope.go:117] "RemoveContainer" containerID="8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f"
Jul 28 20:57:14.600594 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:57:14.600577    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f\": container with ID starting with 8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f not found: ID does not exist" containerID="8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f"
Jul 28 20:57:14.600648 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.600602    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f"} err="failed to get container status \"8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f\": rpc error: code = NotFound desc = could not find container \"8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f\": container with ID starting with 8ce6d6a9a7b269acac51e6020e2ddd1a2e2b05d60622de17775d7a3b2632743f not found: ID does not exist"
Jul 28 20:57:14.600648 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.600620    2582 scope.go:117] "RemoveContainer" containerID="fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb"
Jul 28 20:57:14.600868 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:57:14.600846    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb\": container with ID starting with fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb not found: ID does not exist" containerID="fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb"
Jul 28 20:57:14.600868 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.600874    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb"} err="failed to get container status \"fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb\": rpc error: code = NotFound desc = could not find container \"fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb\": container with ID starting with fd8f44980cb5ea347673677e56c71d97ef46c48d9331f0706b2e56161019e7fb not found: ID does not exist"
Jul 28 20:57:14.603751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.603725    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"]
Jul 28 20:57:14.607650 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:14.607630    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dfc5b-openshift-default-78467658d6-4qhpb"]
Jul 28 20:57:15.444415 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:15.444375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:15.586540 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:15.586506    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:15.715994 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:15.715905    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:15.911477 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:15.911441    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:16.263789 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:16.263753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:16.584148 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:16.584048    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" path="/var/lib/kubelet/pods/7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f/volumes"
Jul 28 20:57:22.454565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:22.454519    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:22.552910 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:22.552866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:22.729905 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:22.729826    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"]
Jul 28 20:57:22.730062 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:22.730038    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy" containerID="cri-o://8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf" gracePeriod=30
Jul 28 20:57:22.742774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:22.742746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:22.989121 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:22.989086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:26.344713 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:26.344664    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:57:26.345133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:26.344748    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:57:27.985124 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:27.985100    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:57:28.106470 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106380    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b478645-fe9a-4429-910f-cbfff26a754c-istio-podinfo\") pod \"3b478645-fe9a-4429-910f-cbfff26a754c\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") "
Jul 28 20:57:28.106470 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106433    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-certs\") pod \"3b478645-fe9a-4429-910f-cbfff26a754c\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") "
Jul 28 20:57:28.106470 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106465    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-data\") pod \"3b478645-fe9a-4429-910f-cbfff26a754c\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") "
Jul 28 20:57:28.106758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106532    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-istio-token\") pod \"3b478645-fe9a-4429-910f-cbfff26a754c\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") "
Jul 28 20:57:28.106758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-socket\") pod \"3b478645-fe9a-4429-910f-cbfff26a754c\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") "
Jul 28 20:57:28.106758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106599    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-credential-socket\") pod \"3b478645-fe9a-4429-910f-cbfff26a754c\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") "
Jul 28 20:57:28.106758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b478645-fe9a-4429-910f-cbfff26a754c-istiod-ca-cert\") pod \"3b478645-fe9a-4429-910f-cbfff26a754c\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") "
Jul 28 20:57:28.106758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106644    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-envoy\") pod \"3b478645-fe9a-4429-910f-cbfff26a754c\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") "
Jul 28 20:57:28.106758 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106668    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdxcp\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-kube-api-access-qdxcp\") pod \"3b478645-fe9a-4429-910f-cbfff26a754c\" (UID: \"3b478645-fe9a-4429-910f-cbfff26a754c\") "
Jul 28 20:57:28.107043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106829    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b478645-fe9a-4429-910f-cbfff26a754c" (UID: "3b478645-fe9a-4429-910f-cbfff26a754c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:28.107043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106908    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b478645-fe9a-4429-910f-cbfff26a754c" (UID: "3b478645-fe9a-4429-910f-cbfff26a754c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:28.107043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.106972    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b478645-fe9a-4429-910f-cbfff26a754c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b478645-fe9a-4429-910f-cbfff26a754c" (UID: "3b478645-fe9a-4429-910f-cbfff26a754c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:57:28.107043 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.107030    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-credential-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:28.107227 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.107048    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:28.107227 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.107115    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b478645-fe9a-4429-910f-cbfff26a754c" (UID: "3b478645-fe9a-4429-910f-cbfff26a754c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:28.107495 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.107470    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b478645-fe9a-4429-910f-cbfff26a754c" (UID: "3b478645-fe9a-4429-910f-cbfff26a754c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:28.108803 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.108779    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-kube-api-access-qdxcp" (OuterVolumeSpecName: "kube-api-access-qdxcp") pod "3b478645-fe9a-4429-910f-cbfff26a754c" (UID: "3b478645-fe9a-4429-910f-cbfff26a754c"). InnerVolumeSpecName "kube-api-access-qdxcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:57:28.108920 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.108857    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b478645-fe9a-4429-910f-cbfff26a754c" (UID: "3b478645-fe9a-4429-910f-cbfff26a754c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:28.109066 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.109046    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b478645-fe9a-4429-910f-cbfff26a754c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b478645-fe9a-4429-910f-cbfff26a754c" (UID: "3b478645-fe9a-4429-910f-cbfff26a754c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:57:28.109131 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.109064    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b478645-fe9a-4429-910f-cbfff26a754c" (UID: "3b478645-fe9a-4429-910f-cbfff26a754c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:57:28.207731 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.207681    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-workload-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:28.207731 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.207725    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b478645-fe9a-4429-910f-cbfff26a754c-istiod-ca-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:28.207731 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.207735    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-envoy\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:28.207731 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.207744    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxcp\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-kube-api-access-qdxcp\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:28.208027 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.207755    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b478645-fe9a-4429-910f-cbfff26a754c-istio-podinfo\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:28.208027 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.207764    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b478645-fe9a-4429-910f-cbfff26a754c-istio-data\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:28.208027 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.207771    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b478645-fe9a-4429-910f-cbfff26a754c-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:28.652399 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.652367    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b478645-fe9a-4429-910f-cbfff26a754c" containerID="8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf" exitCode=0
Jul 28 20:57:28.652579 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.652415    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"
Jul 28 20:57:28.652579 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.652451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" event={"ID":"3b478645-fe9a-4429-910f-cbfff26a754c","Type":"ContainerDied","Data":"8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf"}
Jul 28 20:57:28.652579 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.652507    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7" event={"ID":"3b478645-fe9a-4429-910f-cbfff26a754c","Type":"ContainerDied","Data":"d2ae1c54eed1209c8560baba2ae92500467d3d492616e1cac5d72eccab75b814"}
Jul 28 20:57:28.652579 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.652531    2582 scope.go:117] "RemoveContainer" containerID="8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf"
Jul 28 20:57:28.661402 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.661386    2582 scope.go:117] "RemoveContainer" containerID="5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7"
Jul 28 20:57:28.669999 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.669977    2582 scope.go:117] "RemoveContainer" containerID="8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf"
Jul 28 20:57:28.670270 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:57:28.670252    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf\": container with ID starting with 8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf not found: ID does not exist" containerID="8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf"
Jul 28 20:57:28.670365 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.670279    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf"} err="failed to get container status \"8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf\": rpc error: code = NotFound desc = could not find container \"8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf\": container with ID starting with 8b4f1142b7168da057691c7f1e8b486aec8ce10d4008e853581478593fd4dddf not found: ID does not exist"
Jul 28 20:57:28.670365 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.670323    2582 scope.go:117] "RemoveContainer" containerID="5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7"
Jul 28 20:57:28.670530 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:57:28.670515    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7\": container with ID starting with 5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7 not found: ID does not exist" containerID="5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7"
Jul 28 20:57:28.670577 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.670537    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7"} err="failed to get container status \"5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7\": rpc error: code = NotFound desc = could not find container \"5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7\": container with ID starting with 5cc178892a9a317b2a0ccd061f80ae4dc4851f3edd96b86f2ca6de8d3d169cf7 not found: ID does not exist"
Jul 28 20:57:28.694726 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.694692    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"]
Jul 28 20:57:28.701059 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:28.701027    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ac855-openshift-default-7dc846cc66-fhjm7"]
Jul 28 20:57:30.585994 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.585961    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" path="/var/lib/kubelet/pods/3b478645-fe9a-4429-910f-cbfff26a754c/volumes"
Jul 28 20:57:30.596801 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.596773    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"]
Jul 28 20:57:30.597259 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.597228    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9" podUID="5e98fd13-fc76-4406-9246-6089d3e93b5f" containerName="maas-api" containerID="cri-o://80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f" gracePeriod=30
Jul 28 20:57:30.629930 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.629905    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx"]
Jul 28 20:57:30.635565 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.635538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:30.635689 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.635673    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1b6922-29754525-t57xx"]
Jul 28 20:57:30.712762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.712723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:30.789017 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.788969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:30.859537 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.859507    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:57:30.869331 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:30.869278    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:31.033900 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.033859    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e98fd13-fc76-4406-9246-6089d3e93b5f-openshift-service-ca\") pod \"5e98fd13-fc76-4406-9246-6089d3e93b5f\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") "
Jul 28 20:57:31.034093 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.033922    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5m8s\" (UniqueName: \"kubernetes.io/projected/5e98fd13-fc76-4406-9246-6089d3e93b5f-kube-api-access-j5m8s\") pod \"5e98fd13-fc76-4406-9246-6089d3e93b5f\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") "
Jul 28 20:57:31.034093 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.034062    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e98fd13-fc76-4406-9246-6089d3e93b5f-maas-api-tls\") pod \"5e98fd13-fc76-4406-9246-6089d3e93b5f\" (UID: \"5e98fd13-fc76-4406-9246-6089d3e93b5f\") "
Jul 28 20:57:31.034254 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.034227    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e98fd13-fc76-4406-9246-6089d3e93b5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e98fd13-fc76-4406-9246-6089d3e93b5f" (UID: "5e98fd13-fc76-4406-9246-6089d3e93b5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:57:31.034373 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.034357    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e98fd13-fc76-4406-9246-6089d3e93b5f-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:31.036130 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.036102    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e98fd13-fc76-4406-9246-6089d3e93b5f-kube-api-access-j5m8s" (OuterVolumeSpecName: "kube-api-access-j5m8s") pod "5e98fd13-fc76-4406-9246-6089d3e93b5f" (UID: "5e98fd13-fc76-4406-9246-6089d3e93b5f"). InnerVolumeSpecName "kube-api-access-j5m8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:57:31.036130 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.036114    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e98fd13-fc76-4406-9246-6089d3e93b5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e98fd13-fc76-4406-9246-6089d3e93b5f" (UID: "5e98fd13-fc76-4406-9246-6089d3e93b5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:57:31.135272 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.135184    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5m8s\" (UniqueName: \"kubernetes.io/projected/5e98fd13-fc76-4406-9246-6089d3e93b5f-kube-api-access-j5m8s\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:31.135272 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.135225    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e98fd13-fc76-4406-9246-6089d3e93b5f-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:31.259139 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.259090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:31.683413 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.683372    2582 generic.go:356] "Generic (PLEG): container finished" podID="5e98fd13-fc76-4406-9246-6089d3e93b5f" containerID="80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f" exitCode=0
Jul 28 20:57:31.683914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.683545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9" event={"ID":"5e98fd13-fc76-4406-9246-6089d3e93b5f","Type":"ContainerDied","Data":"80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f"}
Jul 28 20:57:31.683914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.683643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9" event={"ID":"5e98fd13-fc76-4406-9246-6089d3e93b5f","Type":"ContainerDied","Data":"1617fcb5bed238915418b494dfcb298323373800fc9c8d6f145db3e827923086"}
Jul 28 20:57:31.683914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.683687    2582 scope.go:117] "RemoveContainer" containerID="80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f"
Jul 28 20:57:31.683914 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.683866    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"
Jul 28 20:57:31.700341 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.700317    2582 scope.go:117] "RemoveContainer" containerID="80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f"
Jul 28 20:57:31.700648 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:57:31.700628    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f\": container with ID starting with 80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f not found: ID does not exist" containerID="80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f"
Jul 28 20:57:31.700729 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.700658    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f"} err="failed to get container status \"80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f\": rpc error: code = NotFound desc = could not find container \"80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f\": container with ID starting with 80c4b118f221f36f1f604da16a0e64fee79daeddbd2b9f56188ec20727e81e0f not found: ID does not exist"
Jul 28 20:57:31.709793 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.709768    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"]
Jul 28 20:57:31.714845 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:31.714823    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1b6922-66d4c45bdd-pfqm9"]
Jul 28 20:57:32.582620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:32.582518    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e98fd13-fc76-4406-9246-6089d3e93b5f" path="/var/lib/kubelet/pods/5e98fd13-fc76-4406-9246-6089d3e93b5f/volumes"
Jul 28 20:57:32.582924 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:32.582904    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead2bfdb-2439-43a0-a589-08765357e4db" path="/var/lib/kubelet/pods/ead2bfdb-2439-43a0-a589-08765357e4db/volumes"
Jul 28 20:57:33.084563 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:33.084532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:37.818414 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:37.818379    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:37.921553 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:37.921503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:38.094837 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:38.094753    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"]
Jul 28 20:57:38.094988 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:38.094954    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy" containerID="cri-o://11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b" gracePeriod=30
Jul 28 20:57:38.170611 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:38.170565    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:38.345520 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:38.345411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:40.061174 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:40.061127    2582 patch_prober.go:28] interesting pod/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:57:40.061585 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:40.061212    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:57:43.357675 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.357612    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:57:43.448797 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.448760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-data\") pod \"54c698e0-0686-4f6e-a3a8-a919a61eb731\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") "
Jul 28 20:57:43.449007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.448849    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-podinfo\") pod \"54c698e0-0686-4f6e-a3a8-a919a61eb731\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") "
Jul 28 20:57:43.449007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.448883    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-certs\") pod \"54c698e0-0686-4f6e-a3a8-a919a61eb731\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") "
Jul 28 20:57:43.449007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.448921    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-socket\") pod \"54c698e0-0686-4f6e-a3a8-a919a61eb731\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") "
Jul 28 20:57:43.449007 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.448997    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsv89\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-kube-api-access-zsv89\") pod \"54c698e0-0686-4f6e-a3a8-a919a61eb731\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") "
Jul 28 20:57:43.449230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449089    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-credential-socket\") pod \"54c698e0-0686-4f6e-a3a8-a919a61eb731\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") "
Jul 28 20:57:43.449230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449116    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-token\") pod \"54c698e0-0686-4f6e-a3a8-a919a61eb731\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") "
Jul 28 20:57:43.449230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-envoy\") pod \"54c698e0-0686-4f6e-a3a8-a919a61eb731\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") "
Jul 28 20:57:43.449230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449170    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c698e0-0686-4f6e-a3a8-a919a61eb731-istiod-ca-cert\") pod \"54c698e0-0686-4f6e-a3a8-a919a61eb731\" (UID: \"54c698e0-0686-4f6e-a3a8-a919a61eb731\") "
Jul 28 20:57:43.449230 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449213    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54c698e0-0686-4f6e-a3a8-a919a61eb731" (UID: "54c698e0-0686-4f6e-a3a8-a919a61eb731"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:43.449573 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449229    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54c698e0-0686-4f6e-a3a8-a919a61eb731" (UID: "54c698e0-0686-4f6e-a3a8-a919a61eb731"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:43.449573 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449369    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54c698e0-0686-4f6e-a3a8-a919a61eb731" (UID: "54c698e0-0686-4f6e-a3a8-a919a61eb731"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:43.449691 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449596    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-certs\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.449691 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449615    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-workload-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.449691 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449628    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-credential-socket\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.449846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449685    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-data" (OuterVolumeSpecName: "istio-data") pod "54c698e0-0686-4f6e-a3a8-a919a61eb731" (UID: "54c698e0-0686-4f6e-a3a8-a919a61eb731"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:43.449846 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.449711    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c698e0-0686-4f6e-a3a8-a919a61eb731-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54c698e0-0686-4f6e-a3a8-a919a61eb731" (UID: "54c698e0-0686-4f6e-a3a8-a919a61eb731"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:57:43.451203 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.451178    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-kube-api-access-zsv89" (OuterVolumeSpecName: "kube-api-access-zsv89") pod "54c698e0-0686-4f6e-a3a8-a919a61eb731" (UID: "54c698e0-0686-4f6e-a3a8-a919a61eb731"). InnerVolumeSpecName "kube-api-access-zsv89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:57:43.451419 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.451396    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-token" (OuterVolumeSpecName: "istio-token") pod "54c698e0-0686-4f6e-a3a8-a919a61eb731" (UID: "54c698e0-0686-4f6e-a3a8-a919a61eb731"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:57:43.451571 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.451554    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54c698e0-0686-4f6e-a3a8-a919a61eb731" (UID: "54c698e0-0686-4f6e-a3a8-a919a61eb731"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:57:43.451625 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.451588    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54c698e0-0686-4f6e-a3a8-a919a61eb731" (UID: "54c698e0-0686-4f6e-a3a8-a919a61eb731"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:57:43.538318 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.538256    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"]
Jul 28 20:57:43.539582 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.539113    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc" podUID="2c5714ff-3b8d-4672-b674-abed8a46a01c" containerName="maas-api" containerID="cri-o://eceeaecfc0e951460b89673775d787182eaa79aeca47ddb0d5f9c35edac47077" gracePeriod=30
Jul 28 20:57:43.550766 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.550741    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-token\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.550903 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.550769    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-envoy\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.550903 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.550782    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c698e0-0686-4f6e-a3a8-a919a61eb731-istiod-ca-cert\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.550903 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.550792    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-data\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.550903 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.550803    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c698e0-0686-4f6e-a3a8-a919a61eb731-istio-podinfo\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.550903 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.550815    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsv89\" (UniqueName: \"kubernetes.io/projected/54c698e0-0686-4f6e-a3a8-a919a61eb731-kube-api-access-zsv89\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.572786 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.572756    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:43.664887 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.664849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:43.747022 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.746978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:43.752567 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.752539    2582 generic.go:356] "Generic (PLEG): container finished" podID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerID="11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b" exitCode=0
Jul 28 20:57:43.752749 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.752610    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"
Jul 28 20:57:43.752749 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.752622    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" event={"ID":"54c698e0-0686-4f6e-a3a8-a919a61eb731","Type":"ContainerDied","Data":"11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b"}
Jul 28 20:57:43.752749 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.752690    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs" event={"ID":"54c698e0-0686-4f6e-a3a8-a919a61eb731","Type":"ContainerDied","Data":"18458fdc7531176cdd35cabf6be91c9cddd26c0da9c48c72b0437c24607fd134"}
Jul 28 20:57:43.752749 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.752724    2582 scope.go:117] "RemoveContainer" containerID="11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b"
Jul 28 20:57:43.754762 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.754743    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c5714ff-3b8d-4672-b674-abed8a46a01c" containerID="eceeaecfc0e951460b89673775d787182eaa79aeca47ddb0d5f9c35edac47077" exitCode=0
Jul 28 20:57:43.754874 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.754770    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc" event={"ID":"2c5714ff-3b8d-4672-b674-abed8a46a01c","Type":"ContainerDied","Data":"eceeaecfc0e951460b89673775d787182eaa79aeca47ddb0d5f9c35edac47077"}
Jul 28 20:57:43.762092 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.762078    2582 scope.go:117] "RemoveContainer" containerID="0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e"
Jul 28 20:57:43.771139 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.771114    2582 scope.go:117] "RemoveContainer" containerID="11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b"
Jul 28 20:57:43.771448 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:57:43.771429    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b\": container with ID starting with 11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b not found: ID does not exist" containerID="11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b"
Jul 28 20:57:43.771511 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.771458    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b"} err="failed to get container status \"11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b\": rpc error: code = NotFound desc = could not find container \"11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b\": container with ID starting with 11a86dd43c276ba63dc7f7e3303791bb7998a396fe6bc05307fed4022f81f98b not found: ID does not exist"
Jul 28 20:57:43.771511 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.771479    2582 scope.go:117] "RemoveContainer" containerID="0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e"
Jul 28 20:57:43.771723 ip-10-0-128-43 kubenswrapper[2582]: E0728 20:57:43.771709    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e\": container with ID starting with 0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e not found: ID does not exist" containerID="0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e"
Jul 28 20:57:43.771768 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.771727    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e"} err="failed to get container status \"0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e\": rpc error: code = NotFound desc = could not find container \"0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e\": container with ID starting with 0d3a5afa3b1d455e63e4c42e23862f328ca055088aeffd33dec76febb249049e not found: ID does not exist"
Jul 28 20:57:43.775202 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.775174    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"]
Jul 28 20:57:43.779006 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.778985    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-1b6922-openshift-default-c487cb676-jw9hs"]
Jul 28 20:57:43.789827 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.789803    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:57:43.814500 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.814471    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:43.853102 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.853078    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5714ff-3b8d-4672-b674-abed8a46a01c-openshift-service-ca\") pod \"2c5714ff-3b8d-4672-b674-abed8a46a01c\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") "
Jul 28 20:57:43.853249 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.853185    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5714ff-3b8d-4672-b674-abed8a46a01c-maas-api-tls\") pod \"2c5714ff-3b8d-4672-b674-abed8a46a01c\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") "
Jul 28 20:57:43.853249 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.853204    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g8pj\" (UniqueName: \"kubernetes.io/projected/2c5714ff-3b8d-4672-b674-abed8a46a01c-kube-api-access-8g8pj\") pod \"2c5714ff-3b8d-4672-b674-abed8a46a01c\" (UID: \"2c5714ff-3b8d-4672-b674-abed8a46a01c\") "
Jul 28 20:57:43.853473 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.853449    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5714ff-3b8d-4672-b674-abed8a46a01c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c5714ff-3b8d-4672-b674-abed8a46a01c" (UID: "2c5714ff-3b8d-4672-b674-abed8a46a01c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:57:43.855417 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.855382    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5714ff-3b8d-4672-b674-abed8a46a01c-kube-api-access-8g8pj" (OuterVolumeSpecName: "kube-api-access-8g8pj") pod "2c5714ff-3b8d-4672-b674-abed8a46a01c" (UID: "2c5714ff-3b8d-4672-b674-abed8a46a01c"). InnerVolumeSpecName "kube-api-access-8g8pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:57:43.855417 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.855398    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c5714ff-3b8d-4672-b674-abed8a46a01c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c5714ff-3b8d-4672-b674-abed8a46a01c" (UID: "2c5714ff-3b8d-4672-b674-abed8a46a01c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:57:43.954711 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.954631    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5714ff-3b8d-4672-b674-abed8a46a01c-maas-api-tls\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.954711 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.954658    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g8pj\" (UniqueName: \"kubernetes.io/projected/2c5714ff-3b8d-4672-b674-abed8a46a01c-kube-api-access-8g8pj\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:43.954711 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:43.954669    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5714ff-3b8d-4672-b674-abed8a46a01c-openshift-service-ca\") on node \"ip-10-0-128-43.ec2.internal\" DevicePath \"\""
Jul 28 20:57:44.164383 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:44.164351    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:44.583369 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:44.583339    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" path="/var/lib/kubelet/pods/54c698e0-0686-4f6e-a3a8-a919a61eb731/volumes"
Jul 28 20:57:44.761299 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:44.761267    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"
Jul 28 20:57:44.761542 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:44.761275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc" event={"ID":"2c5714ff-3b8d-4672-b674-abed8a46a01c","Type":"ContainerDied","Data":"3cc044686082225d0f47b5456051af91ac6f57adf41ad5a909454463ecb248bd"}
Jul 28 20:57:44.761542 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:44.761374    2582 scope.go:117] "RemoveContainer" containerID="eceeaecfc0e951460b89673775d787182eaa79aeca47ddb0d5f9c35edac47077"
Jul 28 20:57:44.787613 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:44.787564    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"]
Jul 28 20:57:44.789855 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:44.789834    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c0e826-5b7fcbc867-c68fc"]
Jul 28 20:57:46.583259 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:46.583226    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c5714ff-3b8d-4672-b674-abed8a46a01c" path="/var/lib/kubelet/pods/2c5714ff-3b8d-4672-b674-abed8a46a01c/volumes"
Jul 28 20:57:50.592455 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:50.592424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:50.671494 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:50.671462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:50.876893 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:50.876808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:51.110646 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:51.110603    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:57:51.546226 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:51.546189    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ddc577dd-92px5_9ab1c29d-5e53-4efe-bf02-a51a29d71f95/authorino/0.log"
Jul 28 20:57:57.127674 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:57.127642    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pfc69_e947b8e8-7fe7-4950-b024-951a48546a01/manager/0.log"
Jul 28 20:57:57.244448 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:57.244417    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jh9kg_a4e8ff70-c912-489d-80f9-899857887c3b/manager/0.log"
Jul 28 20:57:57.367946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:57.367911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58546478d9-bjkcp_c0d06697-ef11-4db9-bed7-95b1e394199f/manager/0.log"
Jul 28 20:57:57.489586 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:57.489561    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xl6q8_ea9236c2-355d-4eb3-9b21-0234a74bd117/server/0.log"
Jul 28 20:57:57.608670 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:57.608643    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fdbnv_a3328a3b-22dc-490a-a299-3c15ad6ed2ca/manager/0.log"
Jul 28 20:57:58.001968 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:58.001942    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685bc94585-kqxdj_d323113d-b1bc-44d2-9eec-68bb234bd7ec/copy-manifests/0.log"
Jul 28 20:57:58.013761 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:58.013740    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685bc94585-kqxdj_d323113d-b1bc-44d2-9eec-68bb234bd7ec/manager/0.log"
Jul 28 20:57:58.385454 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:58.385378    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74bb79f4fd-7kzlx_d82f1086-427d-4a3c-ba1e-15aa78b38364/maas-api/0.log"
Jul 28 20:57:58.724951 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:58.724878    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754525-mlvps_c3e303bb-1219-434e-9cc9-9ef44eb9d09c/cleanup/0.log"
Jul 28 20:57:58.836839 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:57:58.836805    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wxz7t_e8d74642-814c-4ac1-8b5d-a228d7736d79/postgres/0.log"
Jul 28 20:58:00.217250 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:00.217217    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ddc577dd-92px5_9ab1c29d-5e53-4efe-bf02-a51a29d71f95/authorino/0.log"
Jul 28 20:58:00.337837 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:00.337806    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-h5bzb_31c351c7-2986-4ae1-b9d6-c97d7e95a9da/manager/0.log"
Jul 28 20:58:00.697335 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:00.697286    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cznwc_9c2211be-6d12-4dcc-a703-70d3a2e3e047/registry-server/0.log"
Jul 28 20:58:00.815177 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:00.815144    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7zd55_754eaa7a-c4b9-4d59-8cf6-54499a165e75/manager/0.log"
Jul 28 20:58:01.694356 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:01.694281    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cdf7b948d-9lqhz_9f135a0a-1998-4886-94b9-934315a388c6/kube-auth-proxy/0.log"
Jul 28 20:58:01.936417 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:01.936386    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-c5q2z_2d099fda-4a87-45cb-a88f-a1d546cf1e6e/istio-proxy/1.log"
Jul 28 20:58:02.055192 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:02.055160    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tgj9q_a75904b0-19db-4b4e-86c3-cf7312dd4d0d/payload-pre-processing/0.log"
Jul 28 20:58:02.177657 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:02.177629    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5gbgl_988d2ddd-e609-4125-8a53-b6904e158e7a/payload-processing/0.log"
Jul 28 20:58:02.661700 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:02.661665    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nztjs_095c3f09-0f14-4b09-858e-2dc3202c3c47/main/0.log"
Jul 28 20:58:03.031526 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:03.031495    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl59rx_2f0bd08e-5bb4-4c0b-9872-4aa08fef944f/main/0.log"
Jul 28 20:58:09.255050 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255015    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kdk6l/must-gather-6hh6h"]
Jul 28 20:58:09.255588 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255568    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy"
Jul 28 20:58:09.255671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255592    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy"
Jul 28 20:58:09.255671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255608    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy"
Jul 28 20:58:09.255671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255616    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy"
Jul 28 20:58:09.255671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255628    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e98fd13-fc76-4406-9246-6089d3e93b5f" containerName="maas-api"
Jul 28 20:58:09.255671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255637    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e98fd13-fc76-4406-9246-6089d3e93b5f" containerName="maas-api"
Jul 28 20:58:09.255671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255656    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy"
Jul 28 20:58:09.255671 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255665    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255681    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5714ff-3b8d-4672-b674-abed8a46a01c" containerName="maas-api"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255690    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5714ff-3b8d-4672-b674-abed8a46a01c" containerName="maas-api"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255699    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146d4f9d-b4ab-4b9e-a417-079afc86f224" containerName="maas-api"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255707    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="146d4f9d-b4ab-4b9e-a417-079afc86f224" containerName="maas-api"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255725    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255733    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255742    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255751    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255763    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f3ccc0-a439-415a-a38a-4f98b13a38a8" containerName="revoke-keys"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255771    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f3ccc0-a439-415a-a38a-4f98b13a38a8" containerName="revoke-keys"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255873    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255885    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e98fd13-fc76-4406-9246-6089d3e93b5f" containerName="maas-api"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255894    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255905    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255913    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f3ccc0-a439-415a-a38a-4f98b13a38a8" containerName="revoke-keys"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255923    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="146d4f9d-b4ab-4b9e-a417-079afc86f224" containerName="maas-api"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255958    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255967    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5714ff-3b8d-4672-b674-abed8a46a01c" containerName="maas-api"
Jul 28 20:58:09.256019 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.255978    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b478645-fe9a-4429-910f-cbfff26a754c" containerName="istio-proxy"
Jul 28 20:58:09.256946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.256066    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy"
Jul 28 20:58:09.256946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.256077    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a41f99b-0d0e-4a5c-8e37-7bcfc0576d1f" containerName="istio-proxy"
Jul 28 20:58:09.256946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.256170    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c698e0-0686-4f6e-a3a8-a919a61eb731" containerName="istio-proxy"
Jul 28 20:58:09.259658 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.259637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdk6l/must-gather-6hh6h"
Jul 28 20:58:09.262427 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.262406    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdk6l\"/\"kube-root-ca.crt\""
Jul 28 20:58:09.262508 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.262468    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdk6l\"/\"openshift-service-ca.crt\""
Jul 28 20:58:09.268868 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.268845    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdk6l/must-gather-6hh6h"]
Jul 28 20:58:09.279931 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.279901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:09.376747 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.376718    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f36a77e7-54a5-4213-9401-88dd565d2dc8-must-gather-output\") pod \"must-gather-6hh6h\" (UID: \"f36a77e7-54a5-4213-9401-88dd565d2dc8\") " pod="openshift-must-gather-kdk6l/must-gather-6hh6h"
Jul 28 20:58:09.376925 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.376763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gs8n\" (UniqueName: \"kubernetes.io/projected/f36a77e7-54a5-4213-9401-88dd565d2dc8-kube-api-access-7gs8n\") pod \"must-gather-6hh6h\" (UID: \"f36a77e7-54a5-4213-9401-88dd565d2dc8\") " pod="openshift-must-gather-kdk6l/must-gather-6hh6h"
Jul 28 20:58:09.399861 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.399835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:09.478113 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.478081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gs8n\" (UniqueName: \"kubernetes.io/projected/f36a77e7-54a5-4213-9401-88dd565d2dc8-kube-api-access-7gs8n\") pod \"must-gather-6hh6h\" (UID: \"f36a77e7-54a5-4213-9401-88dd565d2dc8\") " pod="openshift-must-gather-kdk6l/must-gather-6hh6h"
Jul 28 20:58:09.478253 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.478177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f36a77e7-54a5-4213-9401-88dd565d2dc8-must-gather-output\") pod \"must-gather-6hh6h\" (UID: \"f36a77e7-54a5-4213-9401-88dd565d2dc8\") " pod="openshift-must-gather-kdk6l/must-gather-6hh6h"
Jul 28 20:58:09.478519 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.478502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f36a77e7-54a5-4213-9401-88dd565d2dc8-must-gather-output\") pod \"must-gather-6hh6h\" (UID: \"f36a77e7-54a5-4213-9401-88dd565d2dc8\") " pod="openshift-must-gather-kdk6l/must-gather-6hh6h"
Jul 28 20:58:09.490710 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.490679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gs8n\" (UniqueName: \"kubernetes.io/projected/f36a77e7-54a5-4213-9401-88dd565d2dc8-kube-api-access-7gs8n\") pod \"must-gather-6hh6h\" (UID: \"f36a77e7-54a5-4213-9401-88dd565d2dc8\") " pod="openshift-must-gather-kdk6l/must-gather-6hh6h"
Jul 28 20:58:09.569013 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.568934    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdk6l/must-gather-6hh6h"
Jul 28 20:58:09.726027 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.726001    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdk6l/must-gather-6hh6h"]
Jul 28 20:58:09.729370 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:58:09.729338    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf36a77e7_54a5_4213_9401_88dd565d2dc8.slice/crio-83a59c14bb188dc4117736f855d779e146c781a6d8624756b0aa410ef416a8d0 WatchSource:0}: Error finding container 83a59c14bb188dc4117736f855d779e146c781a6d8624756b0aa410ef416a8d0: Status 404 returned error can't find the container with id 83a59c14bb188dc4117736f855d779e146c781a6d8624756b0aa410ef416a8d0
Jul 28 20:58:09.894692 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:09.894594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdk6l/must-gather-6hh6h" event={"ID":"f36a77e7-54a5-4213-9401-88dd565d2dc8","Type":"ContainerStarted","Data":"83a59c14bb188dc4117736f855d779e146c781a6d8624756b0aa410ef416a8d0"}
Jul 28 20:58:10.902844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:10.902763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdk6l/must-gather-6hh6h" event={"ID":"f36a77e7-54a5-4213-9401-88dd565d2dc8","Type":"ContainerStarted","Data":"fe3213557d432ffe0e52ad68aefb0fc6919cfd8a17e43bcb8130cc889fec3f15"}
Jul 28 20:58:10.902844 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:10.902817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdk6l/must-gather-6hh6h" event={"ID":"f36a77e7-54a5-4213-9401-88dd565d2dc8","Type":"ContainerStarted","Data":"2b1037703bb353267cd07577ca1f8b30ba220fd60bbc83f930408ec299892f79"}
Jul 28 20:58:10.922345 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:10.922266    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kdk6l/must-gather-6hh6h" podStartSLOduration=1.010706801 podStartE2EDuration="1.922243071s" totalImagesPullingTime="911.53627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:58:10.92007268 +0000 UTC m=+3540.774383747" watchObservedRunningTime="2026-07-28 20:58:10.922243071 +0000 UTC m=+3540.776554138"
Jul 28 20:58:11.123522 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:11.123484    2582 scope.go:117] "RemoveContainer" containerID="924c4b13a6e68410db2cbbaee74d353d45fe9174b18667936b7a74fc2e9c79db"
Jul 28 20:58:12.293620 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:12.293585    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pcwfl_dedd29f8-8425-42cc-804d-2b786679bd20/global-pull-secret-syncer/0.log"
Jul 28 20:58:12.446451 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:12.446424    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log"
Jul 28 20:58:12.554988 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:12.554895    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-43.ec2.internal_7c858bff10c5634a7a7b495e5272b508/haproxy/0.log"
Jul 28 20:58:17.393312 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:17.390136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ddc577dd-92px5_9ab1c29d-5e53-4efe-bf02-a51a29d71f95/authorino/0.log"
Jul 28 20:58:17.427545 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:17.427513    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-h5bzb_31c351c7-2986-4ae1-b9d6-c97d7e95a9da/manager/0.log"
Jul 28 20:58:17.535067 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:17.535033    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cznwc_9c2211be-6d12-4dcc-a703-70d3a2e3e047/registry-server/0.log"
Jul 28 20:58:17.750311 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:17.747636    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7zd55_754eaa7a-c4b9-4d59-8cf6-54499a165e75/manager/0.log"
Jul 28 20:58:19.446028 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:19.445918    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:20.006835 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:20.006808    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lk77_9084be9d-cb5f-43d9-bac6-187e91e3f73d/node-exporter/0.log"
Jul 28 20:58:20.034829 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:20.034793    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lk77_9084be9d-cb5f-43d9-bac6-187e91e3f73d/kube-rbac-proxy/0.log"
Jul 28 20:58:20.058450 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:20.058421    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lk77_9084be9d-cb5f-43d9-bac6-187e91e3f73d/init-textfile/0.log"
Jul 28 20:58:22.116143 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.116090    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-6tlg5_eb1a1641-bacf-4667-996b-71696f3c7afb/networking-console-plugin/0.log"
Jul 28 20:58:22.212718 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.212676    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"]
Jul 28 20:58:22.220615 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.220589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.224631 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.224590    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kdk6l\"/\"default-dockercfg-cqn65\""
Jul 28 20:58:22.228765 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.228734    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"]
Jul 28 20:58:22.317114 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.317080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-sys\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.317400 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.317375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9x2x\" (UniqueName: \"kubernetes.io/projected/f15798b6-e791-41e3-bcb1-3d73ea234fea-kube-api-access-s9x2x\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.317512 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.317462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-proc\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.317512 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.317492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-lib-modules\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.317623 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.317528    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-podres\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.418227 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.418133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-podres\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.418428 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.418237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-sys\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.418428 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.418283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9x2x\" (UniqueName: \"kubernetes.io/projected/f15798b6-e791-41e3-bcb1-3d73ea234fea-kube-api-access-s9x2x\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.418428 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.418379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-proc\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.418428 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.418404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-lib-modules\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.418639 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.418569    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-lib-modules\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.418698 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.418656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-podres\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.418757 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.418703    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-sys\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.419083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.419055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f15798b6-e791-41e3-bcb1-3d73ea234fea-proc\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.428383 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.428358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9x2x\" (UniqueName: \"kubernetes.io/projected/f15798b6-e791-41e3-bcb1-3d73ea234fea-kube-api-access-s9x2x\") pod \"perf-node-gather-daemonset-cmgts\" (UID: \"f15798b6-e791-41e3-bcb1-3d73ea234fea\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.546074 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.546024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.745814 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.744954    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"]
Jul 28 20:58:22.749542 ip-10-0-128-43 kubenswrapper[2582]: W0728 20:58:22.749501    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf15798b6_e791_41e3_bcb1_3d73ea234fea.slice/crio-e53098a675239ea7441f451a2af150ba94b90a035234409347bcc92298e1722f WatchSource:0}: Error finding container e53098a675239ea7441f451a2af150ba94b90a035234409347bcc92298e1722f: Status 404 returned error can't find the container with id e53098a675239ea7441f451a2af150ba94b90a035234409347bcc92298e1722f
Jul 28 20:58:22.998748 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.998706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts" event={"ID":"f15798b6-e791-41e3-bcb1-3d73ea234fea","Type":"ContainerStarted","Data":"25f062b471b1ebfb8d20597d64658e51481bcad2ac5ca1b9fb400d7abeae22f7"}
Jul 28 20:58:22.998946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.998771    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:22.998946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:22.998788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts" event={"ID":"f15798b6-e791-41e3-bcb1-3d73ea234fea","Type":"ContainerStarted","Data":"e53098a675239ea7441f451a2af150ba94b90a035234409347bcc92298e1722f"}
Jul 28 20:58:23.016493 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:23.016429    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts" podStartSLOduration=1.016405486 podStartE2EDuration="1.016405486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:58:23.015054914 +0000 UTC m=+3552.869365996" watchObservedRunningTime="2026-07-28 20:58:23.016405486 +0000 UTC m=+3552.870716606"
Jul 28 20:58:25.285840 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:25.285815    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sp2jg_5339e904-5d69-4e23-bbc6-32bbda9e309d/dns/0.log"
Jul 28 20:58:25.306458 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:25.306431    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sp2jg_5339e904-5d69-4e23-bbc6-32bbda9e309d/kube-rbac-proxy/0.log"
Jul 28 20:58:25.383133 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:25.383101    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sfwvl_17e3752e-cdc4-4616-b0ce-84df74553189/dns-node-resolver/0.log"
Jul 28 20:58:25.981680 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:25.981651    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-97cb54f69-xgx74_28b6d19d-531d-4167-99d9-a45d465186f1/registry/0.log"
Jul 28 20:58:26.038029 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:26.037996    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-66np6_294f6d80-9bc0-4a40-969a-3801b9c16c3e/node-ca/0.log"
Jul 28 20:58:27.249510 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:27.249478    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cdf7b948d-9lqhz_9f135a0a-1998-4886-94b9-934315a388c6/kube-auth-proxy/0.log"
Jul 28 20:58:27.366280 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:27.366248    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-c5q2z_2d099fda-4a87-45cb-a88f-a1d546cf1e6e/istio-proxy/0.log"
Jul 28 20:58:27.397123 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:27.397091    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-c5q2z_2d099fda-4a87-45cb-a88f-a1d546cf1e6e/istio-proxy/1.log"
Jul 28 20:58:27.430076 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:27.430044    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tgj9q_a75904b0-19db-4b4e-86c3-cf7312dd4d0d/payload-pre-processing/0.log"
Jul 28 20:58:27.464248 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:27.464217    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5gbgl_988d2ddd-e609-4125-8a53-b6904e158e7a/payload-processing/0.log"
Jul 28 20:58:28.370727 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:28.370681    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sm9vj_2da09d65-7fba-4191-ace4-dffe38a24a20/serve-healthcheck-canary/0.log"
Jul 28 20:58:28.937028 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:28.936994    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gzjb6_ec0121bc-2354-4424-9fd2-965012dcfc38/kube-rbac-proxy/0.log"
Jul 28 20:58:28.977719 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:28.977690    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gzjb6_ec0121bc-2354-4424-9fd2-965012dcfc38/exporter/0.log"
Jul 28 20:58:28.999439 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:28.999413    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gzjb6_ec0121bc-2354-4424-9fd2-965012dcfc38/extractor/0.log"
Jul 28 20:58:29.016794 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:29.016764    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-cmgts"
Jul 28 20:58:31.495389 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:31.495359    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pfc69_e947b8e8-7fe7-4950-b024-951a48546a01/manager/0.log"
Jul 28 20:58:31.776083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:31.776005    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jh9kg_a4e8ff70-c912-489d-80f9-899857887c3b/manager/0.log"
Jul 28 20:58:31.980634 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:31.980602    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58546478d9-bjkcp_c0d06697-ef11-4db9-bed7-95b1e394199f/manager/0.log"
Jul 28 20:58:32.054585 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:32.054556    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xl6q8_ea9236c2-355d-4eb3-9b21-0234a74bd117/server/0.log"
Jul 28 20:58:32.195220 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:32.195190    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fdbnv_a3328a3b-22dc-490a-a299-3c15ad6ed2ca/manager/0.log"
Jul 28 20:58:32.431751 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:32.431408    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685bc94585-kqxdj_d323113d-b1bc-44d2-9eec-68bb234bd7ec/manager/0.log"
Jul 28 20:58:32.449760 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:32.449727    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685bc94585-kqxdj_d323113d-b1bc-44d2-9eec-68bb234bd7ec/copy-manifests/0.log"
Jul 28 20:58:33.117527 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:33.117487    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cdf7b948d-9lqhz_9f135a0a-1998-4886-94b9-934315a388c6/kube-auth-proxy/0.log"
Jul 28 20:58:33.154083 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:33.154046    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-c5q2z_2d099fda-4a87-45cb-a88f-a1d546cf1e6e/istio-proxy/0.log"
Jul 28 20:58:33.184223 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:33.184194    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-c5q2z_2d099fda-4a87-45cb-a88f-a1d546cf1e6e/istio-proxy/1.log"
Jul 28 20:58:33.209153 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:33.209125    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tgj9q_a75904b0-19db-4b4e-86c3-cf7312dd4d0d/payload-pre-processing/0.log"
Jul 28 20:58:33.235877 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:33.235849    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5gbgl_988d2ddd-e609-4125-8a53-b6904e158e7a/payload-processing/0.log"
Jul 28 20:58:34.804528 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:34.804498    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cdf7b948d-9lqhz_9f135a0a-1998-4886-94b9-934315a388c6/kube-auth-proxy/0.log"
Jul 28 20:58:34.844172 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:34.844138    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-c5q2z_2d099fda-4a87-45cb-a88f-a1d546cf1e6e/istio-proxy/0.log"
Jul 28 20:58:34.877013 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:34.876975    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-c5q2z_2d099fda-4a87-45cb-a88f-a1d546cf1e6e/istio-proxy/1.log"
Jul 28 20:58:34.904774 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:34.904730    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tgj9q_a75904b0-19db-4b4e-86c3-cf7312dd4d0d/payload-pre-processing/0.log"
Jul 28 20:58:34.931666 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:34.931624    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5gbgl_988d2ddd-e609-4125-8a53-b6904e158e7a/payload-processing/0.log"
Jul 28 20:58:38.616927 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:38.616894    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-t47rm_2508efac-6c60-47b8-9656-9bf464750f3e/migrator/0.log"
Jul 28 20:58:38.645674 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:38.645650    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-t47rm_2508efac-6c60-47b8-9656-9bf464750f3e/graceful-termination/0.log"
Jul 28 20:58:40.173525 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.173492    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7n7lb_efdf8223-ffee-4381-8f46-eb69be388114/kube-multus/0.log"
Jul 28 20:58:40.206096 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.206072    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jmcz_6b9fc684-840f-4a31-8d0c-db439eab595a/kube-multus-additional-cni-plugins/0.log"
Jul 28 20:58:40.228644 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.228612    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jmcz_6b9fc684-840f-4a31-8d0c-db439eab595a/egress-router-binary-copy/0.log"
Jul 28 20:58:40.249804 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.249781    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jmcz_6b9fc684-840f-4a31-8d0c-db439eab595a/cni-plugins/0.log"
Jul 28 20:58:40.274059 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.274034    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jmcz_6b9fc684-840f-4a31-8d0c-db439eab595a/bond-cni-plugin/0.log"
Jul 28 20:58:40.295245 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.295221    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jmcz_6b9fc684-840f-4a31-8d0c-db439eab595a/routeoverride-cni/0.log"
Jul 28 20:58:40.320343 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.320321    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jmcz_6b9fc684-840f-4a31-8d0c-db439eab595a/whereabouts-cni-bincopy/0.log"
Jul 28 20:58:40.342973 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.342949    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jmcz_6b9fc684-840f-4a31-8d0c-db439eab595a/whereabouts-cni/0.log"
Jul 28 20:58:40.816912 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.816870    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nnc2m_6f23cdda-4df1-4276-8197-80943b7648b0/network-metrics-daemon/0.log"
Jul 28 20:58:40.836419 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:40.836393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nnc2m_6f23cdda-4df1-4276-8197-80943b7648b0/kube-rbac-proxy/0.log"
Jul 28 20:58:42.387124 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:42.387098    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9lg5_de3e2f3e-37ae-4b26-867e-d5e30f427b4b/ovn-controller/0.log"
Jul 28 20:58:42.404822 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:42.404796    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9lg5_de3e2f3e-37ae-4b26-867e-d5e30f427b4b/ovn-acl-logging/0.log"
Jul 28 20:58:42.428010 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:42.427973    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9lg5_de3e2f3e-37ae-4b26-867e-d5e30f427b4b/kube-rbac-proxy-node/0.log"
Jul 28 20:58:42.450206 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:42.450174    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9lg5_de3e2f3e-37ae-4b26-867e-d5e30f427b4b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 20:58:42.472125 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:42.472105    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9lg5_de3e2f3e-37ae-4b26-867e-d5e30f427b4b/northd/0.log"
Jul 28 20:58:42.493946 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:42.493924    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9lg5_de3e2f3e-37ae-4b26-867e-d5e30f427b4b/nbdb/0.log"
Jul 28 20:58:42.515084 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:42.515058    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9lg5_de3e2f3e-37ae-4b26-867e-d5e30f427b4b/sbdb/0.log"
Jul 28 20:58:42.586134 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:42.586104    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9lg5_de3e2f3e-37ae-4b26-867e-d5e30f427b4b/ovnkube-controller/0.log"
Jul 28 20:58:43.702514 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:43.702479    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fh5cb_b8f79db1-5f9e-4832-8647-08a142ef88e0/network-check-target-container/0.log"
Jul 28 20:58:44.816820 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:44.816785    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2rgjw_79130341-76cf-448d-9a66-7776250ed070/iptables-alerter/0.log"
Jul 28 20:58:45.634634 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:45.634606    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-pvfwt_9f789c89-5c77-4222-9acf-752bfb1b7994/tuned/0.log"
Jul 28 20:58:49.255553 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:49.255521    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-658d7b7c78-8s5w2_8bda1dd0-3f04-4e68-b64d-8878c5203ab7/service-ca-controller/0.log"
Jul 28 20:58:49.682449 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:49.682370    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6b4th_61fa245c-f0d0-44e9-8077-da9a73f46d35/csi-driver/0.log"
Jul 28 20:58:49.705193 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:49.705127    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6b4th_61fa245c-f0d0-44e9-8077-da9a73f46d35/csi-node-driver-registrar/0.log"
Jul 28 20:58:49.754633 ip-10-0-128-43 kubenswrapper[2582]: I0728 20:58:49.754605    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6b4th_61fa245c-f0d0-44e9-8077-da9a73f46d35/csi-liveness-probe/0.log"
Jul 28 21:00:11.171568 ip-10-0-128-43 kubenswrapper[2582]: I0728 21:00:11.171536    2582 scope.go:117] "RemoveContainer" containerID="2bedfd1ffa99b3cc8c03f3def4d0a49fe8d8c77de9930acbd0354c47e6588af6"
Jul 28 21:00:30.720278 ip-10-0-128-43 kubenswrapper[2582]: I0728 21:00:30.720242    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 21:00:33.110670 ip-10-0-128-43 kubenswrapper[2582]: I0728 21:00:33.110584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024
Jul 28 21:00:38.726925 ip-10-0-128-43 kubenswrapper[2582]: I0728 21:00:38.726869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bz87_ec723a1d-bb69-4939-814a-23074980c3bb/konnectivity-agent/0.log" limit=1024