Jul 13 21:09:11.789329 ip-10-0-141-152 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 21:09:12.232914 ip-10-0-141-152 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 21:09:12.232914 ip-10-0-141-152 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 21:09:12.232914 ip-10-0-141-152 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 21:09:12.232914 ip-10-0-141-152 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 21:09:12.232914 ip-10-0-141-152 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 21:09:12.235887 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.235805    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 21:09:12.241943 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241922    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:09:12.241943 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241939    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:09:12.241943 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241942    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:09:12.241943 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241947    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:09:12.241943 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241950    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241953    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241957    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241959    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241962    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241965    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241968    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241971    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241974    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241976    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241979    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241982    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241985    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241988    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241990    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241993    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241995    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.241998    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242000    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242003    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:09:12.242160 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242005    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242008    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242010    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242013    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242015    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242017    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242023    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242027    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242029    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242032    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242034    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242037    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242039    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242069    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242077    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242082    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242085    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242087    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242090    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242093    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:09:12.242631 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242096    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242099    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242102    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242105    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242108    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242110    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242114    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242116    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242119    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242121    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242124    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242126    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242129    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242131    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242134    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242136    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242139    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242142    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242144    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242146    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:09:12.243146 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242148    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242151    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242154    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242156    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242159    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242161    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242166    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242168    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242172    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242177    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242181    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242184    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242187    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242189    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242192    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242195    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242201    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242205    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242207    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:09:12.243607 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242210    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242213    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242216    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242218    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242221    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242223    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242226    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242228    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242231    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242233    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242236    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242238    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242241    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242245    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242248    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242252    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242972    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242981    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242984    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242987    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:09:12.244070 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242990    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242993    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242996    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.242998    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243001    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243003    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243006    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243008    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243011    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243013    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243016    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243018    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243021    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243024    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243026    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243028    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243031    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243033    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243036    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243038    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243060    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:09:12.244671 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243063    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243066    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243068    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243071    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243073    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243076    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243078    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243081    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243083    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243086    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243089    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243092    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243095    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243097    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243100    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243102    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243105    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243107    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243110    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:09:12.245236 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243112    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243114    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243117    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243119    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243122    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243124    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243128    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243131    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243133    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243135    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243138    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243141    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243143    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243146    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243148    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243152    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243156    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243158    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243162    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:09:12.245694 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243164    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243167    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243169    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243172    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243175    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243178    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243181    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243184    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243186    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243189    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243191    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243193    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243196    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243198    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243201    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243203    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243206    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243208    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243211    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:09:12.246194 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243215    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243219    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243222    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243224    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243227    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243230    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243232    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243235    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243239    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243242    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243245    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243248    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243250    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243253    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243256    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243259    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243261    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243647    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243653    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243656    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:09:12.246650 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243658    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243661    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243663    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243666    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243668    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243671    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243674    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243676    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243679    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243681    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243684    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243686    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243689    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243692    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243694    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243697    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243699    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243702    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243704    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243707    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:09:12.247134 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243712    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243715    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243717    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243720    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243722    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243725    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243729    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243733    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243736    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243739    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243742    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243744    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243747    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243750    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243753    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243756    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243759    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243762    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243764    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:09:12.247609 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243767    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243769    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243772    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243774    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243777    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243781    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243785    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243788    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243791    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243793    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243796    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243800    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243802    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243806    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243808    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243811    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243813    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243816    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243818    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243821    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:09:12.248064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243824    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243826    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243829    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243831    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243833    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243836    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243840    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243844    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243846    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243849    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243852    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243855    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243858    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243860    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243863    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243866    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243869    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243871    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243874    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:09:12.248539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243877    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243880    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243883    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243886    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243890    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243892    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243896    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243899    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243901    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243904    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243906    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243909    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243911    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243913    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243916    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243918    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243920    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.243923    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.243997    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244004    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244015    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 21:09:12.248997 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244019    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244023    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244026    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244030    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244034    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244037    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244056    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244063    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244066    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244069    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244072    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244075    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244078    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244082    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244085    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244089    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244092    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244095    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244098    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244102    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244105    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244108    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244111    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 21:09:12.249509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244114    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244117    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244119    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244122    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244125    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244127    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244131    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244134    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244137    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244139    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244142    2564 flags.go:64] FLAG: --enable-server="true"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244144    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244149    2564 flags.go:64] FLAG: --event-burst="100"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244151    2564 flags.go:64] FLAG: --event-qps="50"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244154    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244157    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244159    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244163    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244166    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244168    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244171    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244173    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244176    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244183    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244186    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 21:09:12.250080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244188    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244191    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244194    2564 flags.go:64] FLAG: --feature-gates=""
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244198    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244201    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244204    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244207    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244210    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244213    2564 flags.go:64] FLAG: --help="false"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244215    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244218    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244221    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244224    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244227    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244230    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244232    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244235    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244237    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244240    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244243    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244246    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244249    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244251    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244254    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 21:09:12.250747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244257    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244259    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244262    2564 flags.go:64] FLAG: --lock-file=""
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244264    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244267    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244270    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244275    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244299    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244303    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244306    2564 flags.go:64] FLAG: --logging-format="text"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244309    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244312    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244315    2564 flags.go:64] FLAG: --manifest-url=""
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244318    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244322    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244324    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244328    2564 flags.go:64] FLAG: --max-pods="110"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244331    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244333    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244336    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244339    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244341    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244344    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244347    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244353    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 21:09:12.251333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244356    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244359    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244362    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244365    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244370    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244373    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244375    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244378    2564 flags.go:64] FLAG: --port="10250"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244381    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244383    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02eda528116c92a4d"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244386    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244389    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244392    2564 flags.go:64] FLAG: --register-node="true"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244394    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244397    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244400    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244403    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244406    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244409    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244412    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244415    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244417    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244420    2564 flags.go:64] FLAG: --runonce="false"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244423    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244425    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 21:09:12.251929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244428    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244431    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244433    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244436    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244439    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244442    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244444    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244447    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244450    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244454    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244457    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244460    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244462    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244467    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244470    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244473    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244476    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244479    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244482    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244484    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244487    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244489    2564 flags.go:64] FLAG: --v="2"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244493    2564 flags.go:64] FLAG: --version="false"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244497    2564 flags.go:64] FLAG: --vmodule=""
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244502    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 21:09:12.252539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.244505    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244634    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244637    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244641    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244644    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244647    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244649    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244652    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244655    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244658    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244660    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244663    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244666    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244668    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244671    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244675    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244679    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244681    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244684    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244687    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:09:12.253139 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244690    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244693    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244695    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244697    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244700    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244703    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244705    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244707    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244710    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244712    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244714    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244721    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244724    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244727    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244729    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244733    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244736    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244739    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244742    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:09:12.253623 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244745    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244748    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244751    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244753    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244755    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244758    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244761    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244763    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244766    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244770    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244773    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244775    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244778    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244780    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244782    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244785    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244789    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244792    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244795    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:09:12.254149 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244798    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244800    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244802    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244805    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244807    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244811    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244814    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244816    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244819    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244821    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244823    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244827    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244830    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244833    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244836    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244839    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244841    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244844    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244847    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244849    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:09:12.254598 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244851    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244854    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244858    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244860    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244862    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244865    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244867    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244869    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244872    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244874    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244877    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244879    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244881    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244884    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244886    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244888    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244891    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244894    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244897    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244906    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:09:12.255178 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244908    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.244911    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.245793    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.252795    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.252811    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.253788    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.253795    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253854    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253860    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253863    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253866    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253869    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253871    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253874    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253876    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:09:12.255667 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253879    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253881    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253884    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253887    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253889    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253892    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253894    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253896    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253899    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253901    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253904    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253907    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253909    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253912    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253914    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253917    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253920    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253923    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253925    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:09:12.256028 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253927    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253930    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253934    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253936    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253939    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253941    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253944    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253946    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253949    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253951    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253953    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253956    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253958    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253961    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253963    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253965    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253968    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253970    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253973    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253975    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:09:12.257251 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253978    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253981    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253985    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253989    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253991    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253995    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.253998    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254002    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254005    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254008    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254011    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254013    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254016    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254019    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254022    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254025    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254028    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254030    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254033    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254036    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:09:12.257722 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254038    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254062    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254065    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254067    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254070    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254072    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254075    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254077    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254080    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254082    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254085    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254087    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254090    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254092    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254094    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254097    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254099    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254102    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254104    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:09:12.258269 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254107    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254109    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254112    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254114    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254118    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254120    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254123    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254125    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254128    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254130    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254133    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254135    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254137    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.254142    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254259    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:09:12.258720 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254263    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254267    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254270    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254273    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254276    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254279    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254282    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254284    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254287    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254289    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254292    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254294    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254298    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254301    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254304    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254307    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254310    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254312    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254315    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:09:12.259158 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254317    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254320    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254322    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254325    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254328    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254330    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254333    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254335    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254337    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254339    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254342    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254344    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254346    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254349    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254352    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254354    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254356    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254359    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254361    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254364    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:09:12.259612 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254366    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254368    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254371    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254373    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254376    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254378    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254380    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254383    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254385    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254388    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254390    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254393    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254395    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254397    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254400    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254403    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254406    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254409    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254411    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254414    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:09:12.260087 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254416    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254418    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254421    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254423    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254426    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254428    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254430    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254432    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254435    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254438    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254440    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254442    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254444    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254447    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254449    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254451    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254454    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254456    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254458    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254461    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:09:12.260550 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254463    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254469    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254472    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254474    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254477    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254479    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254482    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254484    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254487    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254490    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254492    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254495    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254497    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254500    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254503    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254506    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254509    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254511    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:09:12.261009 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:12.254513    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:09:12.261446 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.254518    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 21:09:12.261446 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.254608    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 21:09:12.261446 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.256796    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 21:09:12.261446 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.258356    2564 server.go:1019] "Starting client certificate rotation"
Jul 13 21:09:12.261446 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.258458    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 21:09:12.261446 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.259415    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 21:09:12.261446 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.261268    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 21:09:12.264498 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.264481    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 21:09:12.274771 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.274741    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 13 21:09:12.277799 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.277779    2564 log.go:25] "Validated CRI v1 image API"
Jul 13 21:09:12.278587 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.278571    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 21:09:12.281318 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.281300    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 822c8e2b-1fe6-4941-b5a4-951b3cf5ac7f:/dev/nvme0n1p3 d51a0fc9-f73d-4d89-8abb-cf9e73edb8e4:/dev/nvme0n1p4]
Jul 13 21:09:12.281369 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.281318    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 21:09:12.287352 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.287251    2564 manager.go:216] Machine: {Timestamp:2026-07-13 21:09:12.285792096 +0000 UTC m=+0.370988009 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103915 MemoryCapacity:32812183552 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28ba4dfcf00ed3ba9ffe71966d6626 SystemUUID:ec28ba4d-fcf0-0ed3-ba9f-fe71966d6626 BootID:b0036ccf-0fb2-4505-901b-ccc1c25a667f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406093824 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c8:13:2d:e6:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c8:13:2d:e6:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:b3:b4:c3:13:d9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812183552 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 21:09:12.287352 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.287347    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 21:09:12.287455 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.287443    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 21:09:12.289670 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.289654    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 21:09:12.291073 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.291030    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 21:09:12.291195 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.291076    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-152.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 21:09:12.291907 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.291897    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 21:09:12.291937 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.291911    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 21:09:12.291937 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.291917    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 21:09:12.292529 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.292519    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 21:09:12.294321 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.294311    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 21:09:12.297866 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.297855    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 21:09:12.297986 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.297977    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 21:09:12.298018 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.298008    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 21:09:12.298065 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.298020    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 21:09:12.298065 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.298034    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 21:09:12.298118 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.298069    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 13 21:09:12.298118 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.298082    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 21:09:12.298400 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.298389    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 21:09:12.298443 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.298407    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 21:09:12.300844 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.300830    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 21:09:12.303163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.303151    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 21:09:12.303205 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.303181    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 21:09:12.303324 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.303314    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 21:09:12.303355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.303329    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 21:09:12.303355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.303335    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 21:09:12.303355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.303340    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 21:09:12.303355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.303348    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 21:09:12.303355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.303353    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 21:09:12.304011 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.304003    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 21:09:12.304011 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.304011    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 21:09:12.304082 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.304017    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 21:09:12.304082 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.304022    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 21:09:12.304082 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.304030    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 21:09:12.304082 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.304062    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 21:09:12.304954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.304946    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 21:09:12.304954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.304955    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 21:09:12.307999 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.307987    2564 server.go:1268] "Started kubelet"
Jul 13 21:09:12.308100 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.308074    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 21:09:12.308168 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.308087    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 21:09:12.308168 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.308149    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 21:09:12.310929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.310907    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-152.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 21:09:12.310929 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.310918    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 21:09:12.311067 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.310698    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 21:09:12.311090 ip-10-0-141-152 systemd[1]: Started Kubernetes Kubelet.
Jul 13 21:09:12.312260 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.312240    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 21:09:12.313833 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.313819    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 21:09:12.316949 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.315854    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-152.ec2.internal.18c1f59611ccd0d5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-152.ec2.internal,UID:ip-10-0-141-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-152.ec2.internal,},FirstTimestamp:2026-07-13 21:09:12.307962069 +0000 UTC m=+0.393157981,LastTimestamp:2026-07-13 21:09:12.307962069 +0000 UTC m=+0.393157981,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-152.ec2.internal,}"
Jul 13 21:09:12.317226 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.317207    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 21:09:12.320224 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.320203    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 21:09:12.320224 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.320205    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 21:09:12.320376 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.320333    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:12.321000 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.320979    2564 factory.go:55] Registering systemd factory
Jul 13 21:09:12.321000 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.320998    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 13 21:09:12.321157 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321075    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 21:09:12.321157 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321089    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 21:09:12.321157 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321090    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 21:09:12.321289 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321229    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 21:09:12.321289 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321238    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 21:09:12.321366 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321313    2564 factory.go:153] Registering CRI-O factory
Jul 13 21:09:12.321366 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321325    2564 factory.go:223] Registration of the crio container factory successfully
Jul 13 21:09:12.321457 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321371    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 21:09:12.321457 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321393    2564 factory.go:103] Registering Raw factory
Jul 13 21:09:12.321457 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321404    2564 manager.go:1196] Started watching for new ooms in manager
Jul 13 21:09:12.321714 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.321703    2564 manager.go:349] Starting recovery of all containers
Jul 13 21:09:12.323915 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.323885    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 21:09:12.324018 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.323990    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 21:09:12.330713 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.330610    2564 manager.go:354] Recovery completed
Jul 13 21:09:12.335398 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.335379    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vvk8r"
Jul 13 21:09:12.336448 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.336434    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:09:12.338229 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.338207    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:09:12.338297 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.338240    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:09:12.338297 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.338252    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:09:12.339430 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.339415    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 21:09:12.339430 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.339430    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 21:09:12.339531 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.339422    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:09:12.339531 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.339446    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 21:09:12.339873 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.339816    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-152.ec2.internal.18c1f596139aa157  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-152.ec2.internal,UID:ip-10-0-141-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-152.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-152.ec2.internal,},FirstTimestamp:2026-07-13 21:09:12.338227543 +0000 UTC m=+0.423423463,LastTimestamp:2026-07-13 21:09:12.338227543 +0000 UTC m=+0.423423463,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-152.ec2.internal,}"
Jul 13 21:09:12.340167 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.340150    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:09:12.340212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.340184    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:09:12.340212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.340199    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:09:12.340583 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.340572    2564 policy_none.go:49] "None policy: Start"
Jul 13 21:09:12.340621 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.340588    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 21:09:12.340621 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.340605    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 21:09:12.341756 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.341746    2564 policy_none.go:47] "Start"
Jul 13 21:09:12.342655 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.342640    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vvk8r"
Jul 13 21:09:12.377589 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.377561    2564 manager.go:337] "Starting Device Plugin manager"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.377596    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.377609    2564 server.go:87] "Starting device plugin registration server"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.377812    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.377822    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.377887    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.377998    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.378007    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.378435    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.378474    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.386637    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.387858    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.387894    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.387913    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 21:09:12.388070 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.387941    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 21:09:12.390477 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.390459    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:09:12.478422 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.478405    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:09:12.479282 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.479265    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:09:12.479368 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.479296    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:09:12.479368 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.479310    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:09:12.479368 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.479345    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.488312 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.488264    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.488312 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.488287    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-152.ec2.internal\": node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:12.488434 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.488364    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal"]
Jul 13 21:09:12.488434 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.488428    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:09:12.489606 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.489300    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:09:12.489606 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.489325    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:09:12.489606 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.489339    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:09:12.490551 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.490536    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:09:12.490697 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.490673    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.490749 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.490702    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:09:12.491239 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.491223    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:09:12.491239 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.491234    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:09:12.491356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.491253    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:09:12.491356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.491267    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:09:12.491356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.491253    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:09:12.491356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.491326    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:09:12.492858 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.492844    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.492920 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.492871    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:09:12.493508 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.493493    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:09:12.493575 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.493521    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:09:12.493575 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.493531    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:09:12.514031 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.514012    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:12.518543 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.518530    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-152.ec2.internal\" not found" node="ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.522679 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.522655    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bd17d048e9092b3fa303749ed84aa38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal\" (UID: \"8bd17d048e9092b3fa303749ed84aa38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.522754 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.522692    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bd17d048e9092b3fa303749ed84aa38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal\" (UID: \"8bd17d048e9092b3fa303749ed84aa38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.522754 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.522708    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-152.ec2.internal\" not found" node="ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.522754 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.522715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05a87461878d507ac253066a80c27967-config\") pod \"kube-apiserver-proxy-ip-10-0-141-152.ec2.internal\" (UID: \"05a87461878d507ac253066a80c27967\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.614509 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.614484    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:12.622858 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.622840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bd17d048e9092b3fa303749ed84aa38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal\" (UID: \"8bd17d048e9092b3fa303749ed84aa38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.622912 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.622865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bd17d048e9092b3fa303749ed84aa38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal\" (UID: \"8bd17d048e9092b3fa303749ed84aa38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.622912 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.622883    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05a87461878d507ac253066a80c27967-config\") pod \"kube-apiserver-proxy-ip-10-0-141-152.ec2.internal\" (UID: \"05a87461878d507ac253066a80c27967\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.622977 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.622922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05a87461878d507ac253066a80c27967-config\") pod \"kube-apiserver-proxy-ip-10-0-141-152.ec2.internal\" (UID: \"05a87461878d507ac253066a80c27967\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.622977 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.622954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bd17d048e9092b3fa303749ed84aa38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal\" (UID: \"8bd17d048e9092b3fa303749ed84aa38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.622977 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.622954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bd17d048e9092b3fa303749ed84aa38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal\" (UID: \"8bd17d048e9092b3fa303749ed84aa38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.714992 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.714973    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:12.815735 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.815719    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:12.820906 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.820892    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.826894 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:12.826872    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:12.916216 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:12.916192    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:13.016709 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.016684    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:13.117288 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.117234    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-152.ec2.internal\" not found"
Jul 13 21:09:13.185058 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.185031    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:09:13.220932 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.220909    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:13.235561 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.235541    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 21:09:13.237826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.237813    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal"
Jul 13 21:09:13.246302 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.246290    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 21:09:13.257438 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.257421    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 21:09:13.257554 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.257533    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 21:09:13.257603 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.257567    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 21:09:13.298719 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.298704    2564 apiserver.go:52] "Watching apiserver"
Jul 13 21:09:13.320318 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.320298    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 21:09:13.321424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.320661    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 21:09:13.321424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.321104    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-d4sdl","kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5","openshift-image-registry/node-ca-kcjdj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal","openshift-multus/multus-additional-cni-plugins-xwxvt","openshift-multus/multus-j7r2w","kube-system/global-pull-secret-syncer-xl4b6","openshift-cluster-node-tuning-operator/tuned-78cbw","openshift-dns/node-resolver-s4zzn","openshift-multus/network-metrics-daemon-fwj6w","openshift-network-diagnostics/network-check-target-8jrfk","openshift-network-operator/iptables-alerter-wj8pg","openshift-ovn-kubernetes/ovnkube-node-mx6cw"]
Jul 13 21:09:13.322700 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.322682    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.323752 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.323729    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.324955 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.324937    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.325251 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.325232    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 21:09:13.325360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.325291    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 21:09:13.325360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.325300    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 21:09:13.325459 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.325368    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gqnh6\""
Jul 13 21:09:13.325752 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.325739    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 21:09:13.326138 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.326121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.326382 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.326361    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 21:09:13.326631 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.326617    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 21:09:13.326748 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.326736    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqlwk\""
Jul 13 21:09:13.327370 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.327353    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:13.327894 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.327879    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 21:09:13.327954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.327925    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 21:09:13.327987 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.327925    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 21:09:13.328291 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.328278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:13.328346 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.328329    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:13.329006 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.328993    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 21:09:13.329375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.329357    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6q2bl\""
Jul 13 21:09:13.329451 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.329366    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-flzzf\""
Jul 13 21:09:13.329451 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.329401    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.329553 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.329470    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 21:09:13.329658 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.329645    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 21:09:13.330256 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.330242    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 21:09:13.330335 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.330299    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qh4tz\""
Jul 13 21:09:13.330473 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.330455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.330582 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.330567    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 21:09:13.331730 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.331714    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:13.331968 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.331764    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:13.331968 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.331795    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvqn8\""
Jul 13 21:09:13.331968 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.331820    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 21:09:13.332105 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.332017    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 21:09:13.332867 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.332848    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 21:09:13.332867 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.332907    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 21:09:13.333156 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.332957    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p7v8n\""
Jul 13 21:09:13.333375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.333362    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:13.333473 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.333417    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:13.333599 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.333584    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 21:09:13.334389 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.334374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.337724 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.335660    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.338438 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.338419    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 21:09:13.338671 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.338650    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 21:09:13.338794 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.338774    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5vgv9\""
Jul 13 21:09:13.338909 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.338785    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 21:09:13.338909 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.338802    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 21:09:13.338909 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.338858    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z5wh6\""
Jul 13 21:09:13.338909 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.338892    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 21:09:13.339258 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.339239    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 21:09:13.339329 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.339278    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 21:09:13.339383 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.339368    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 21:09:13.339485 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.339468    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 21:09:13.346655 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.346612    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 21:04:13 +0000 UTC" deadline="2028-02-29 18:42:26.433129688 +0000 UTC"
Jul 13 21:09:13.346655 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.346653    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14301h33m13.086478751s"
Jul 13 21:09:13.358488 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.358468    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xpj6c"
Jul 13 21:09:13.372900 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.372884    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xpj6c"
Jul 13 21:09:13.421976 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.421958    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 21:09:13.425383 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-run-ovn\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.425490 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.425490 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425425    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7173a77a-9b2f-4837-81d5-0863d79a9008-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.425490 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5233d551-7dd2-41f4-b820-99367023e3b4-tmp\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.425490 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30ac7b1f-d387-4c5a-9b4a-3c99304f8d45-hosts-file\") pod \"node-resolver-s4zzn\" (UID: \"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45\") " pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.425647 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdbg\" (UniqueName: \"kubernetes.io/projected/30ac7b1f-d387-4c5a-9b4a-3c99304f8d45-kube-api-access-5zdbg\") pod \"node-resolver-s4zzn\" (UID: \"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45\") " pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.425647 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:13.425647 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425546    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5fk\" (UniqueName: \"kubernetes.io/projected/40ff0e81-61a7-4cd6-8f39-d973f557e974-kube-api-access-gw5fk\") pod \"iptables-alerter-wj8pg\" (UID: \"40ff0e81-61a7-4cd6-8f39-d973f557e974\") " pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.425647 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425570    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff20edcf-5053-44fb-af08-bc34f7a74a49-ovnkube-config\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.425647 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.425647 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425612    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-sys\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.425647 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/166204bf-e576-42fa-80df-02d81b311427-dbus\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425654    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a07bee79-0d6e-439f-b920-be4cd37930b3-cni-binary-copy\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-var-lib-cni-multus\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425702    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzx4n\" (UniqueName: \"kubernetes.io/projected/ff20edcf-5053-44fb-af08-bc34f7a74a49-kube-api-access-kzx4n\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425725    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99b9abac-1432-4aec-b1c7-3e2550422583-agent-certs\") pod \"konnectivity-agent-d4sdl\" (UID: \"99b9abac-1432-4aec-b1c7-3e2550422583\") " pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-modprobe-d\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-var-lib-kubelet\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-log-socket\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425824    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbll\" (UniqueName: \"kubernetes.io/projected/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-kube-api-access-rjbll\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76f5b2b-3323-4299-9eef-aa9bc1763402-host\") pod \"node-ca-kcjdj\" (UID: \"f76f5b2b-3323-4299-9eef-aa9bc1763402\") " pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425884    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-system-cni-dir\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl4z\" (UniqueName: \"kubernetes.io/projected/5233d551-7dd2-41f4-b820-99367023e3b4-kube-api-access-gwl4z\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-os-release\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.425971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-var-lib-cni-bin\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.425996    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40ff0e81-61a7-4cd6-8f39-d973f557e974-host-slash\") pod \"iptables-alerter-wj8pg\" (UID: \"40ff0e81-61a7-4cd6-8f39-d973f557e974\") " pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426020    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff20edcf-5053-44fb-af08-bc34f7a74a49-ovn-node-metrics-cert\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426098    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff20edcf-5053-44fb-af08-bc34f7a74a49-ovnkube-script-lib\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-registration-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426138    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-socket-dir-parent\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426158    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-var-lib-kubelet\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426178    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbh9m\" (UniqueName: \"kubernetes.io/projected/a07bee79-0d6e-439f-b920-be4cd37930b3-kube-api-access-zbh9m\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426211    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-run-systemd\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnbt\" (UniqueName: \"kubernetes.io/projected/7173a77a-9b2f-4837-81d5-0863d79a9008-kube-api-access-bxnbt\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426253    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-systemd\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-run\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30ac7b1f-d387-4c5a-9b4a-3c99304f8d45-tmp-dir\") pod \"node-resolver-s4zzn\" (UID: \"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45\") " pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426315    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-systemd-units\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-sysconfig\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.426418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-run-netns\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-run-multus-certs\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426401    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-host\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-cni-dir\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426460    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-slash\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426504    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7173a77a-9b2f-4837-81d5-0863d79a9008-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426548    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99b9abac-1432-4aec-b1c7-3e2550422583-konnectivity-ca\") pod \"konnectivity-agent-d4sdl\" (UID: \"99b9abac-1432-4aec-b1c7-3e2550422583\") " pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426569    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5233d551-7dd2-41f4-b820-99367023e3b4-etc-tuned\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4wd\" (UniqueName: \"kubernetes.io/projected/6e598283-d5e4-489e-a446-3a4fca30fb56-kube-api-access-mk4wd\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-system-cni-dir\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-run-k8s-cni-cncf-io\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426675    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-conf-dir\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426707    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-kubelet\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.426966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426782    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-socket-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7173a77a-9b2f-4837-81d5-0863d79a9008-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-node-log\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff20edcf-5053-44fb-af08-bc34f7a74a49-env-overrides\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-sys-fs\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-cnibin\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-daemon-config\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.426976    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427002    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-var-lib-openvswitch\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-etc-openvswitch\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-cni-netd\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vskrd\" (UniqueName: \"kubernetes.io/projected/f76f5b2b-3323-4299-9eef-aa9bc1763402-kube-api-access-vskrd\") pod \"node-ca-kcjdj\" (UID: \"f76f5b2b-3323-4299-9eef-aa9bc1763402\") " pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-os-release\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-sysctl-conf\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427178    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/40ff0e81-61a7-4cd6-8f39-d973f557e974-iptables-alerter-script\") pod \"iptables-alerter-wj8pg\" (UID: \"40ff0e81-61a7-4cd6-8f39-d973f557e974\") " pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-etc-kubernetes\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.427721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/166204bf-e576-42fa-80df-02d81b311427-kubelet-config\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-run-netns\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-cni-bin\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76f5b2b-3323-4299-9eef-aa9bc1763402-serviceca\") pod \"node-ca-kcjdj\" (UID: \"f76f5b2b-3323-4299-9eef-aa9bc1763402\") " pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427347    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-kubernetes\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-sysctl-d\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427403    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-cnibin\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-run-openvswitch\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-device-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-lib-modules\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.428499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.427494    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-hostroot\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.441369 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.441341    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a87461878d507ac253066a80c27967.slice/crio-aca20dff5cd7ee290e94929aef4d381c1cc1f23187b84cdf3312fb2d1630e803 WatchSource:0}: Error finding container aca20dff5cd7ee290e94929aef4d381c1cc1f23187b84cdf3312fb2d1630e803: Status 404 returned error can't find the container with id aca20dff5cd7ee290e94929aef4d381c1cc1f23187b84cdf3312fb2d1630e803
Jul 13 21:09:13.441558 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.441536    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd17d048e9092b3fa303749ed84aa38.slice/crio-099af095c31d4072909c1667cf1127545d14d00ec1e579929a4c09dc1248a1b8 WatchSource:0}: Error finding container 099af095c31d4072909c1667cf1127545d14d00ec1e579929a4c09dc1248a1b8: Status 404 returned error can't find the container with id 099af095c31d4072909c1667cf1127545d14d00ec1e579929a4c09dc1248a1b8
Jul 13 21:09:13.446243 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.446229    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:09:13.527949 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.527922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-run-netns\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.528064 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.527951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-run-multus-certs\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.528064 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.527967    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:13.528064 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.527987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-host\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.528064 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-cni-dir\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.528064 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-run-multus-certs\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.528064 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-slash\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-run-netns\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528092    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-host\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528104    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-slash\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528097    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528120    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-cni-dir\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7173a77a-9b2f-4837-81d5-0863d79a9008-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528175    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99b9abac-1432-4aec-b1c7-3e2550422583-konnectivity-ca\") pod \"konnectivity-agent-d4sdl\" (UID: \"99b9abac-1432-4aec-b1c7-3e2550422583\") " pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5233d551-7dd2-41f4-b820-99367023e3b4-etc-tuned\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4wd\" (UniqueName: \"kubernetes.io/projected/6e598283-d5e4-489e-a446-3a4fca30fb56-kube-api-access-mk4wd\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.528124    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-system-cni-dir\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528290    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-run-k8s-cni-cncf-io\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.528323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528327    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528340    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-run-k8s-cni-cncf-io\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.528356    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret podName:166204bf-e576-42fa-80df-02d81b311427 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:14.028335268 +0000 UTC m=+2.113531170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret") pod "global-pull-secret-syncer-xl4b6" (UID: "166204bf-e576-42fa-80df-02d81b311427") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-system-cni-dir\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-conf-dir\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-kubelet\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528445    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-conf-dir\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528455    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-socket-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528477    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-kubelet\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7173a77a-9b2f-4837-81d5-0863d79a9008-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-node-log\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528520    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528531    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff20edcf-5053-44fb-af08-bc34f7a74a49-env-overrides\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-sys-fs\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528569    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-socket-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-sys-fs\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-node-log\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-cnibin\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-daemon-config\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528683    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528727    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-var-lib-openvswitch\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99b9abac-1432-4aec-b1c7-3e2550422583-konnectivity-ca\") pod \"konnectivity-agent-d4sdl\" (UID: \"99b9abac-1432-4aec-b1c7-3e2550422583\") " pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-etc-openvswitch\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-cni-netd\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528811    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-cnibin\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vskrd\" (UniqueName: \"kubernetes.io/projected/f76f5b2b-3323-4299-9eef-aa9bc1763402-kube-api-access-vskrd\") pod \"node-ca-kcjdj\" (UID: \"f76f5b2b-3323-4299-9eef-aa9bc1763402\") " pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-os-release\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528914    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-os-release\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528958    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-etc-openvswitch\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.528998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-cni-netd\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff20edcf-5053-44fb-af08-bc34f7a74a49-env-overrides\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-sysctl-conf\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/40ff0e81-61a7-4cd6-8f39-d973f557e974-iptables-alerter-script\") pod \"iptables-alerter-wj8pg\" (UID: \"40ff0e81-61a7-4cd6-8f39-d973f557e974\") " pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7173a77a-9b2f-4837-81d5-0863d79a9008-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.529788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-etc-kubernetes\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/166204bf-e576-42fa-80df-02d81b311427-kubelet-config\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-run-netns\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529190    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-etc-kubernetes\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-cni-bin\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76f5b2b-3323-4299-9eef-aa9bc1763402-serviceca\") pod \"node-ca-kcjdj\" (UID: \"f76f5b2b-3323-4299-9eef-aa9bc1763402\") " pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/166204bf-e576-42fa-80df-02d81b311427-kubelet-config\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-var-lib-openvswitch\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-sysctl-conf\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-kubernetes\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-daemon-config\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529286    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-sysctl-d\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-kubernetes\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529296    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-cni-bin\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7173a77a-9b2f-4837-81d5-0863d79a9008-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-run-netns\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-cnibin\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-cnibin\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.530689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529365    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-sysctl-d\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529367    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-run-openvswitch\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-run-openvswitch\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-device-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-lib-modules\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529571    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76f5b2b-3323-4299-9eef-aa9bc1763402-serviceca\") pod \"node-ca-kcjdj\" (UID: \"f76f5b2b-3323-4299-9eef-aa9bc1763402\") " pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-hostroot\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-device-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529616    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-run-ovn\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529622    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-hostroot\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529643    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-run-ovn\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7173a77a-9b2f-4837-81d5-0863d79a9008-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529678    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-lib-modules\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5233d551-7dd2-41f4-b820-99367023e3b4-tmp\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529693    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30ac7b1f-d387-4c5a-9b4a-3c99304f8d45-hosts-file\") pod \"node-resolver-s4zzn\" (UID: \"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45\") " pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdbg\" (UniqueName: \"kubernetes.io/projected/30ac7b1f-d387-4c5a-9b4a-3c99304f8d45-kube-api-access-5zdbg\") pod \"node-resolver-s4zzn\" (UID: \"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45\") " pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.531766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5fk\" (UniqueName: \"kubernetes.io/projected/40ff0e81-61a7-4cd6-8f39-d973f557e974-kube-api-access-gw5fk\") pod \"iptables-alerter-wj8pg\" (UID: \"40ff0e81-61a7-4cd6-8f39-d973f557e974\") " pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529787    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30ac7b1f-d387-4c5a-9b4a-3c99304f8d45-hosts-file\") pod \"node-resolver-s4zzn\" (UID: \"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45\") " pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff20edcf-5053-44fb-af08-bc34f7a74a49-ovnkube-config\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-sys\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/166204bf-e576-42fa-80df-02d81b311427-dbus\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.529896    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a07bee79-0d6e-439f-b920-be4cd37930b3-cni-binary-copy\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.529952    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs podName:6e598283-d5e4-489e-a446-3a4fca30fb56 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:14.029936739 +0000 UTC m=+2.115132652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs") pod "network-metrics-daemon-fwj6w" (UID: "6e598283-d5e4-489e-a446-3a4fca30fb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.529981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-var-lib-cni-multus\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzx4n\" (UniqueName: \"kubernetes.io/projected/ff20edcf-5053-44fb-af08-bc34f7a74a49-kube-api-access-kzx4n\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99b9abac-1432-4aec-b1c7-3e2550422583-agent-certs\") pod \"konnectivity-agent-d4sdl\" (UID: \"99b9abac-1432-4aec-b1c7-3e2550422583\") " pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-modprobe-d\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530127    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-var-lib-kubelet\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-log-socket\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.532672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530191    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbll\" (UniqueName: \"kubernetes.io/projected/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-kube-api-access-rjbll\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530208    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-sys\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76f5b2b-3323-4299-9eef-aa9bc1763402-host\") pod \"node-ca-kcjdj\" (UID: \"f76f5b2b-3323-4299-9eef-aa9bc1763402\") " pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530217    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/40ff0e81-61a7-4cd6-8f39-d973f557e974-iptables-alerter-script\") pod \"iptables-alerter-wj8pg\" (UID: \"40ff0e81-61a7-4cd6-8f39-d973f557e974\") " pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-system-cni-dir\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl4z\" (UniqueName: \"kubernetes.io/projected/5233d551-7dd2-41f4-b820-99367023e3b4-kube-api-access-gwl4z\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-os-release\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530311    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-var-lib-cni-bin\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40ff0e81-61a7-4cd6-8f39-d973f557e974-host-slash\") pod \"iptables-alerter-wj8pg\" (UID: \"40ff0e81-61a7-4cd6-8f39-d973f557e974\") " pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530359    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-var-lib-cni-multus\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530388    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff20edcf-5053-44fb-af08-bc34f7a74a49-ovn-node-metrics-cert\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a07bee79-0d6e-439f-b920-be4cd37930b3-cni-binary-copy\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530413    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff20edcf-5053-44fb-af08-bc34f7a74a49-ovnkube-script-lib\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-registration-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530461    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-socket-dir-parent\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530477    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff20edcf-5053-44fb-af08-bc34f7a74a49-ovnkube-config\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.533308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-var-lib-kubelet\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbh9m\" (UniqueName: \"kubernetes.io/projected/a07bee79-0d6e-439f-b920-be4cd37930b3-kube-api-access-zbh9m\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-run-systemd\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnbt\" (UniqueName: \"kubernetes.io/projected/7173a77a-9b2f-4837-81d5-0863d79a9008-kube-api-access-bxnbt\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530583    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-systemd\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-modprobe-d\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530335    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/166204bf-e576-42fa-80df-02d81b311427-dbus\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-run\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530661    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76f5b2b-3323-4299-9eef-aa9bc1763402-host\") pod \"node-ca-kcjdj\" (UID: \"f76f5b2b-3323-4299-9eef-aa9bc1763402\") " pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530684    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530696    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-var-lib-cni-bin\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40ff0e81-61a7-4cd6-8f39-d973f557e974-host-slash\") pod \"iptables-alerter-wj8pg\" (UID: \"40ff0e81-61a7-4cd6-8f39-d973f557e974\") " pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-run\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7173a77a-9b2f-4837-81d5-0863d79a9008-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530740    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-var-lib-kubelet\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530808    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7173a77a-9b2f-4837-81d5-0863d79a9008-system-cni-dir\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530816    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-os-release\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30ac7b1f-d387-4c5a-9b4a-3c99304f8d45-tmp-dir\") pod \"node-resolver-s4zzn\" (UID: \"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45\") " pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.533793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530847    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-run-systemd\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530869    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-systemd-units\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530883    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-systemd\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530897    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-sysconfig\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530914    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-host-var-lib-kubelet\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.530990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5233d551-7dd2-41f4-b820-99367023e3b4-etc-sysconfig\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.531090    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-registration-dir\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.531152    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a07bee79-0d6e-439f-b920-be4cd37930b3-multus-socket-dir-parent\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.531174    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-log-socket\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.531233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff20edcf-5053-44fb-af08-bc34f7a74a49-systemd-units\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.531284    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30ac7b1f-d387-4c5a-9b4a-3c99304f8d45-tmp-dir\") pod \"node-resolver-s4zzn\" (UID: \"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45\") " pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.531384    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff20edcf-5053-44fb-af08-bc34f7a74a49-ovnkube-script-lib\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.531855    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5233d551-7dd2-41f4-b820-99367023e3b4-etc-tuned\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.532692    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5233d551-7dd2-41f4-b820-99367023e3b4-tmp\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.533001    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99b9abac-1432-4aec-b1c7-3e2550422583-agent-certs\") pod \"konnectivity-agent-d4sdl\" (UID: \"99b9abac-1432-4aec-b1c7-3e2550422583\") " pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:13.534288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.533146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff20edcf-5053-44fb-af08-bc34f7a74a49-ovn-node-metrics-cert\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.555037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.555019    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzx4n\" (UniqueName: \"kubernetes.io/projected/ff20edcf-5053-44fb-af08-bc34f7a74a49-kube-api-access-kzx4n\") pod \"ovnkube-node-mx6cw\" (UID: \"ff20edcf-5053-44fb-af08-bc34f7a74a49\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.563307 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.563286    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:09:13.563393 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.563314    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:09:13.563393 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.563329    2564 projected.go:196] Error preparing data for projected volume kube-api-access-jtnnq for pod openshift-network-diagnostics/network-check-target-8jrfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:13.563393 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:13.563392    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq podName:a2a289c2-21cf-445d-897e-9c004b7b0ba4 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:14.063377472 +0000 UTC m=+2.148573387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jtnnq" (UniqueName: "kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq") pod "network-check-target-8jrfk" (UID: "a2a289c2-21cf-445d-897e-9c004b7b0ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:13.565461 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.565439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdbg\" (UniqueName: \"kubernetes.io/projected/30ac7b1f-d387-4c5a-9b4a-3c99304f8d45-kube-api-access-5zdbg\") pod \"node-resolver-s4zzn\" (UID: \"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45\") " pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.565601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.565516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnbt\" (UniqueName: \"kubernetes.io/projected/7173a77a-9b2f-4837-81d5-0863d79a9008-kube-api-access-bxnbt\") pod \"multus-additional-cni-plugins-xwxvt\" (UID: \"7173a77a-9b2f-4837-81d5-0863d79a9008\") " pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.565698 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.565614    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vskrd\" (UniqueName: \"kubernetes.io/projected/f76f5b2b-3323-4299-9eef-aa9bc1763402-kube-api-access-vskrd\") pod \"node-ca-kcjdj\" (UID: \"f76f5b2b-3323-4299-9eef-aa9bc1763402\") " pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.566302 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.566275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5fk\" (UniqueName: \"kubernetes.io/projected/40ff0e81-61a7-4cd6-8f39-d973f557e974-kube-api-access-gw5fk\") pod \"iptables-alerter-wj8pg\" (UID: \"40ff0e81-61a7-4cd6-8f39-d973f557e974\") " pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.567375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.567357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbll\" (UniqueName: \"kubernetes.io/projected/5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9-kube-api-access-rjbll\") pod \"aws-ebs-csi-driver-node-wz6t5\" (UID: \"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.567444 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.567431    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl4z\" (UniqueName: \"kubernetes.io/projected/5233d551-7dd2-41f4-b820-99367023e3b4-kube-api-access-gwl4z\") pod \"tuned-78cbw\" (UID: \"5233d551-7dd2-41f4-b820-99367023e3b4\") " pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.567901 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.567887    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbh9m\" (UniqueName: \"kubernetes.io/projected/a07bee79-0d6e-439f-b920-be4cd37930b3-kube-api-access-zbh9m\") pod \"multus-j7r2w\" (UID: \"a07bee79-0d6e-439f-b920-be4cd37930b3\") " pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.567942 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.567892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4wd\" (UniqueName: \"kubernetes.io/projected/6e598283-d5e4-489e-a446-3a4fca30fb56-kube-api-access-mk4wd\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:13.648269 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.648208    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5"
Jul 13 21:09:13.656420 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.656398    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5e8d7f_b0d8_4709_b439_9f08f86fb0e9.slice/crio-0e205ac363fe9b375c73c829406c57bf4826c67bb0fbc29ac4d24132cb8196de WatchSource:0}: Error finding container 0e205ac363fe9b375c73c829406c57bf4826c67bb0fbc29ac4d24132cb8196de: Status 404 returned error can't find the container with id 0e205ac363fe9b375c73c829406c57bf4826c67bb0fbc29ac4d24132cb8196de
Jul 13 21:09:13.665336 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.665320    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcjdj"
Jul 13 21:09:13.667002 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.666983    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j7r2w"
Jul 13 21:09:13.670649 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.670629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xwxvt"
Jul 13 21:09:13.673281 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.673261    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76f5b2b_3323_4299_9eef_aa9bc1763402.slice/crio-75c0394c8d7d56dd458f526e02a189af2ae87d719d2fde9cabad2ecd85bbd61a WatchSource:0}: Error finding container 75c0394c8d7d56dd458f526e02a189af2ae87d719d2fde9cabad2ecd85bbd61a: Status 404 returned error can't find the container with id 75c0394c8d7d56dd458f526e02a189af2ae87d719d2fde9cabad2ecd85bbd61a
Jul 13 21:09:13.676914 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.676868    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07bee79_0d6e_439f_b920_be4cd37930b3.slice/crio-457b554a897d4dd01a0b46f52cebde531ef2c5145511a4821e3d74c3cd172e31 WatchSource:0}: Error finding container 457b554a897d4dd01a0b46f52cebde531ef2c5145511a4821e3d74c3cd172e31: Status 404 returned error can't find the container with id 457b554a897d4dd01a0b46f52cebde531ef2c5145511a4821e3d74c3cd172e31
Jul 13 21:09:13.680316 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.680299    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7173a77a_9b2f_4837_81d5_0863d79a9008.slice/crio-896f92f28e300c53a24038e46ad6845758ee6eacd8994a72a1e78e4ff03750a7 WatchSource:0}: Error finding container 896f92f28e300c53a24038e46ad6845758ee6eacd8994a72a1e78e4ff03750a7: Status 404 returned error can't find the container with id 896f92f28e300c53a24038e46ad6845758ee6eacd8994a72a1e78e4ff03750a7
Jul 13 21:09:13.692768 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.692751    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:09:13.693779 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.693765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:13.700877 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.700854    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b9abac_1432_4aec_b1c7_3e2550422583.slice/crio-47e8b17e5913d88ecc03fc497e5ab0fcf3d3cd8c69dddf5429e51459bae7ca67 WatchSource:0}: Error finding container 47e8b17e5913d88ecc03fc497e5ab0fcf3d3cd8c69dddf5429e51459bae7ca67: Status 404 returned error can't find the container with id 47e8b17e5913d88ecc03fc497e5ab0fcf3d3cd8c69dddf5429e51459bae7ca67
Jul 13 21:09:13.711644 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.711628    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-78cbw"
Jul 13 21:09:13.717887 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.717865    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5233d551_7dd2_41f4_b820_99367023e3b4.slice/crio-c1f50658110f53d22904a00a81513f6a89cb94407bd2b4250fbeb5b064fb868e WatchSource:0}: Error finding container c1f50658110f53d22904a00a81513f6a89cb94407bd2b4250fbeb5b064fb868e: Status 404 returned error can't find the container with id c1f50658110f53d22904a00a81513f6a89cb94407bd2b4250fbeb5b064fb868e
Jul 13 21:09:13.725130 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.725113    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4zzn"
Jul 13 21:09:13.731321 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.731298    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ac7b1f_d387_4c5a_9b4a_3c99304f8d45.slice/crio-c7d9722aaedfe677e73b243cf6c2fd179b040effc9cc75bf4018300564b4963f WatchSource:0}: Error finding container c7d9722aaedfe677e73b243cf6c2fd179b040effc9cc75bf4018300564b4963f: Status 404 returned error can't find the container with id c7d9722aaedfe677e73b243cf6c2fd179b040effc9cc75bf4018300564b4963f
Jul 13 21:09:13.745421 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.745392    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wj8pg"
Jul 13 21:09:13.750401 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:13.750385    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:13.752431 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.752413    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ff0e81_61a7_4cd6_8f39_d973f557e974.slice/crio-26745d6e9bca1e9a855b457798570130667ff34f5bc54959ead5c56a358a1703 WatchSource:0}: Error finding container 26745d6e9bca1e9a855b457798570130667ff34f5bc54959ead5c56a358a1703: Status 404 returned error can't find the container with id 26745d6e9bca1e9a855b457798570130667ff34f5bc54959ead5c56a358a1703
Jul 13 21:09:13.756629 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:13.756610    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff20edcf_5053_44fb_af08_bc34f7a74a49.slice/crio-72afd05e1150e07e2af62996a626f62ab2eb560f3ee875d18c0cbc810bbaad18 WatchSource:0}: Error finding container 72afd05e1150e07e2af62996a626f62ab2eb560f3ee875d18c0cbc810bbaad18: Status 404 returned error can't find the container with id 72afd05e1150e07e2af62996a626f62ab2eb560f3ee875d18c0cbc810bbaad18
Jul 13 21:09:15.298521 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.298505    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:09:15.462529 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.462474    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:15.462529 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.462511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:15.462691 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:15.462616    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:15.462691 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:15.462616    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:15.462691 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:15.462668    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs podName:6e598283-d5e4-489e-a446-3a4fca30fb56 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:16.462653465 +0000 UTC m=+3.120310449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs") pod "network-metrics-daemon-fwj6w" (UID: "6e598283-d5e4-489e-a446-3a4fca30fb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:15.462691 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:15.462683    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret podName:166204bf-e576-42fa-80df-02d81b311427 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:16.462675234 +0000 UTC m=+3.120332217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret") pod "global-pull-secret-syncer-xl4b6" (UID: "166204bf-e576-42fa-80df-02d81b311427") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:15.563289 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.563261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:15.563456 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:15.563440    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:09:15.563544 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:15.563462    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:09:15.563544 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:15.563474    2564 projected.go:196] Error preparing data for projected volume kube-api-access-jtnnq for pod openshift-network-diagnostics/network-check-target-8jrfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:15.563544 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:15.563525    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq podName:a2a289c2-21cf-445d-897e-9c004b7b0ba4 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:16.563507221 +0000 UTC m=+3.221164211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtnnq" (UniqueName: "kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq") pod "network-check-target-8jrfk" (UID: "a2a289c2-21cf-445d-897e-9c004b7b0ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:15.802377 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.802305    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:04:14 +0000 UTC" deadline="2028-03-14 14:49:01.035340076 +0000 UTC"
Jul 13 21:09:15.802377 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.802374    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14633h39m45.2329696s"
Jul 13 21:09:15.818711 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.818685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:15.818870 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:15.818846    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:15.828840 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.828790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5" event={"ID":"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9","Type":"ContainerStarted","Data":"0e205ac363fe9b375c73c829406c57bf4826c67bb0fbc29ac4d24132cb8196de"}
Jul 13 21:09:15.835543 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.835505    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal" event={"ID":"05a87461878d507ac253066a80c27967","Type":"ContainerStarted","Data":"aca20dff5cd7ee290e94929aef4d381c1cc1f23187b84cdf3312fb2d1630e803"}
Jul 13 21:09:15.842075 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.842029    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4zzn" event={"ID":"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45","Type":"ContainerStarted","Data":"c7d9722aaedfe677e73b243cf6c2fd179b040effc9cc75bf4018300564b4963f"}
Jul 13 21:09:15.846488 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.846457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-78cbw" event={"ID":"5233d551-7dd2-41f4-b820-99367023e3b4","Type":"ContainerStarted","Data":"c1f50658110f53d22904a00a81513f6a89cb94407bd2b4250fbeb5b064fb868e"}
Jul 13 21:09:15.848815 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.848789    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwxvt" event={"ID":"7173a77a-9b2f-4837-81d5-0863d79a9008","Type":"ContainerStarted","Data":"896f92f28e300c53a24038e46ad6845758ee6eacd8994a72a1e78e4ff03750a7"}
Jul 13 21:09:15.857686 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.857660    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal" event={"ID":"8bd17d048e9092b3fa303749ed84aa38","Type":"ContainerStarted","Data":"099af095c31d4072909c1667cf1127545d14d00ec1e579929a4c09dc1248a1b8"}
Jul 13 21:09:15.860255 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.860230    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerStarted","Data":"72afd05e1150e07e2af62996a626f62ab2eb560f3ee875d18c0cbc810bbaad18"}
Jul 13 21:09:15.862368 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.862326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wj8pg" event={"ID":"40ff0e81-61a7-4cd6-8f39-d973f557e974","Type":"ContainerStarted","Data":"26745d6e9bca1e9a855b457798570130667ff34f5bc54959ead5c56a358a1703"}
Jul 13 21:09:15.865305 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.865249    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d4sdl" event={"ID":"99b9abac-1432-4aec-b1c7-3e2550422583","Type":"ContainerStarted","Data":"47e8b17e5913d88ecc03fc497e5ab0fcf3d3cd8c69dddf5429e51459bae7ca67"}
Jul 13 21:09:15.867410 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.867350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j7r2w" event={"ID":"a07bee79-0d6e-439f-b920-be4cd37930b3","Type":"ContainerStarted","Data":"457b554a897d4dd01a0b46f52cebde531ef2c5145511a4821e3d74c3cd172e31"}
Jul 13 21:09:15.873066 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:15.873016    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcjdj" event={"ID":"f76f5b2b-3323-4299-9eef-aa9bc1763402","Type":"ContainerStarted","Data":"75c0394c8d7d56dd458f526e02a189af2ae87d719d2fde9cabad2ecd85bbd61a"}
Jul 13 21:09:16.260815 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:16.260787    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:09:16.472825 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:16.472788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:16.473003 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:16.472860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:16.473061 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.473005    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:16.473115 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.473071    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret podName:166204bf-e576-42fa-80df-02d81b311427 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:18.473052082 +0000 UTC m=+5.130709084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret") pod "global-pull-secret-syncer-xl4b6" (UID: "166204bf-e576-42fa-80df-02d81b311427") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:16.473471 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.473451    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:16.473550 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.473517    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs podName:6e598283-d5e4-489e-a446-3a4fca30fb56 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:18.473497568 +0000 UTC m=+5.131154571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs") pod "network-metrics-daemon-fwj6w" (UID: "6e598283-d5e4-489e-a446-3a4fca30fb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:16.573922 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:16.573840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:16.574087 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.574063    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:09:16.574145 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.574101    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:09:16.574145 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.574116    2564 projected.go:196] Error preparing data for projected volume kube-api-access-jtnnq for pod openshift-network-diagnostics/network-check-target-8jrfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:16.574248 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.574179    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq podName:a2a289c2-21cf-445d-897e-9c004b7b0ba4 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:18.574159695 +0000 UTC m=+5.231816690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtnnq" (UniqueName: "kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq") pod "network-check-target-8jrfk" (UID: "a2a289c2-21cf-445d-897e-9c004b7b0ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:16.803462 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:16.803386    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:04:14 +0000 UTC" deadline="2028-01-29 23:06:15.729159855 +0000 UTC"
Jul 13 21:09:16.803462 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:16.803428    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13561h56m58.925736487s"
Jul 13 21:09:16.816652 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:16.815941    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:16.816652 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.816070    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:16.816652 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:16.816478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:16.816652 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:16.816603    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:17.337818 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:17.337345    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:09:17.819191 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:17.818640    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:17.819191 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:17.818753    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:18.489127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:18.489091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:18.489326 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:18.489162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:18.489326 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.489282    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:18.489437 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.489346    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret podName:166204bf-e576-42fa-80df-02d81b311427 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:22.489327145 +0000 UTC m=+9.146984138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret") pod "global-pull-secret-syncer-xl4b6" (UID: "166204bf-e576-42fa-80df-02d81b311427") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:18.489786 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.489768    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:18.489876 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.489825    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs podName:6e598283-d5e4-489e-a446-3a4fca30fb56 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:22.489807867 +0000 UTC m=+9.147464859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs") pod "network-metrics-daemon-fwj6w" (UID: "6e598283-d5e4-489e-a446-3a4fca30fb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:18.590207 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:18.590112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:18.599140 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.593932    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:09:18.599140 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.593969    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:09:18.599140 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.593995    2564 projected.go:196] Error preparing data for projected volume kube-api-access-jtnnq for pod openshift-network-diagnostics/network-check-target-8jrfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:18.599140 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.594070    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq podName:a2a289c2-21cf-445d-897e-9c004b7b0ba4 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:22.594038313 +0000 UTC m=+9.251695306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtnnq" (UniqueName: "kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq") pod "network-check-target-8jrfk" (UID: "a2a289c2-21cf-445d-897e-9c004b7b0ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:18.816212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:18.816133    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:18.816366 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.816264    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:18.816366 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:18.816344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:18.816487 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:18.816425    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:19.816734 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:19.816703    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:19.817223 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:19.816817    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:20.816522 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:20.815889    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:20.816522 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:20.816015    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:20.816522 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:20.816396    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:20.816522 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:20.816474    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:21.816679 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:21.816644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:21.816856 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:21.816766    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:22.518570 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:22.518482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:22.518570 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:22.518561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:22.518789 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.518692    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:22.518789 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.518737    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs podName:6e598283-d5e4-489e-a446-3a4fca30fb56 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:30.518723027 +0000 UTC m=+17.176380011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs") pod "network-metrics-daemon-fwj6w" (UID: "6e598283-d5e4-489e-a446-3a4fca30fb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:22.518789 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.518777    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:22.518890 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.518796    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret podName:166204bf-e576-42fa-80df-02d81b311427 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:30.518790037 +0000 UTC m=+17.176447021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret") pod "global-pull-secret-syncer-xl4b6" (UID: "166204bf-e576-42fa-80df-02d81b311427") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:22.619348 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:22.619312    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:22.619513 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.619474    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:09:22.619513 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.619497    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:09:22.619513 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.619511    2564 projected.go:196] Error preparing data for projected volume kube-api-access-jtnnq for pod openshift-network-diagnostics/network-check-target-8jrfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:22.619674 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.619572    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq podName:a2a289c2-21cf-445d-897e-9c004b7b0ba4 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:30.619551761 +0000 UTC m=+17.277208759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtnnq" (UniqueName: "kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq") pod "network-check-target-8jrfk" (UID: "a2a289c2-21cf-445d-897e-9c004b7b0ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:22.815821 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:22.815748    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:22.815959 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.815892    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:22.816297 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:22.816278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:22.816401 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:22.816381    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:23.817748 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:23.817715    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:23.818378 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:23.818329    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:24.815945 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:24.815910    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:24.816125 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:24.816042    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:24.816125 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:24.816086    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:24.816244 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:24.816222    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:25.816248 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:25.816216    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:25.816735 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:25.816345    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:26.816272 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:26.816241    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:26.816635 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:26.816348    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:26.816635 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:26.816376    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:26.816635 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:26.816425    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:27.815969 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:27.815929    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:27.816155 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:27.816055    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:28.816047 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:28.816012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:28.816469 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:28.816144    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:28.816469 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:28.816013    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:28.816608 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:28.816521    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:29.816420 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:29.816386    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:29.816845 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:29.816518    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:30.577151 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:30.577113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:30.577367 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:30.577187    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:30.577367 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.577298    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:30.577367 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.577357    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret podName:166204bf-e576-42fa-80df-02d81b311427 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:46.57733636 +0000 UTC m=+33.234993345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret") pod "global-pull-secret-syncer-xl4b6" (UID: "166204bf-e576-42fa-80df-02d81b311427") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:30.577836 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.577495    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:30.577836 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.577558    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs podName:6e598283-d5e4-489e-a446-3a4fca30fb56 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:46.577538346 +0000 UTC m=+33.235195367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs") pod "network-metrics-daemon-fwj6w" (UID: "6e598283-d5e4-489e-a446-3a4fca30fb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:30.678104 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:30.678069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:30.678269 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.678215    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:09:30.678269 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.678243    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:09:30.678269 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.678256    2564 projected.go:196] Error preparing data for projected volume kube-api-access-jtnnq for pod openshift-network-diagnostics/network-check-target-8jrfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:30.678378 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.678317    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq podName:a2a289c2-21cf-445d-897e-9c004b7b0ba4 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:46.678299014 +0000 UTC m=+33.335956021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtnnq" (UniqueName: "kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq") pod "network-check-target-8jrfk" (UID: "a2a289c2-21cf-445d-897e-9c004b7b0ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:30.815717 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:30.815690    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:30.815905 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.815793    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:30.815905 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:30.815856    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:30.816016 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:30.815964    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:31.816011 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:31.815976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:31.816493 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:31.816090    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:32.816680 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:32.816649    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:32.816680 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:32.816666    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:32.817080 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:32.816752    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:32.817080 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:32.816828    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:33.817056 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:33.817015    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:33.817492 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:33.817132    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:34.816125 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:34.816090    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:34.816286 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:34.816221    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:34.816342 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:34.816278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:34.816412 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:34.816396    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:35.816444 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:35.816420    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:35.816717 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:35.816517    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:35.936967 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:35.936939    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal" event={"ID":"05a87461878d507ac253066a80c27967","Type":"ContainerStarted","Data":"a2eb39b15df7df0714761c858f5523f550deca23cd702ad1ec47cf95988f1eb5"}
Jul 13 21:09:36.816846 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.816572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:36.817294 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.816572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:36.817294 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:36.816946    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:36.817294 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:36.816988    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:36.941233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.941208    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:09:36.941566 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.941546    2564 generic.go:356] "Generic (PLEG): container finished" podID="ff20edcf-5053-44fb-af08-bc34f7a74a49" containerID="1f0efe8e474c65a78382a913dfb21ece61f602a58e85686052c13f0a8e231c30" exitCode=1
Jul 13 21:09:36.941747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.941628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerStarted","Data":"741abcb1acae01890e6678c3901220090591f86ffb287bd70bf681881a77d1b3"}
Jul 13 21:09:36.941747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.941714    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerStarted","Data":"c22ea0c2d6c1071dc20430273a16cfed6b8371026ca63ef99e4cde9e194f35a6"}
Jul 13 21:09:36.941858 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.941765    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerStarted","Data":"c1f80b57f11e7417e5f979bb36a99e80f7b5280073306a0bceed600c5676f61c"}
Jul 13 21:09:36.941858 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.941815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerStarted","Data":"9529fff4c9fee973695132d957de0ed8686bf536a9280210b8730de6c25badb9"}
Jul 13 21:09:36.941954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.941865    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerDied","Data":"1f0efe8e474c65a78382a913dfb21ece61f602a58e85686052c13f0a8e231c30"}
Jul 13 21:09:36.941954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.941916    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerStarted","Data":"6132128fe20a8cea41afd1fcb9f591afc5a2ce347a43518b9ee29be9d5d460b7"}
Jul 13 21:09:36.943102 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.943076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d4sdl" event={"ID":"99b9abac-1432-4aec-b1c7-3e2550422583","Type":"ContainerStarted","Data":"1f5d058de21db963c0b9bea736beae00c980aecb6603731f9f922ff2bb03a35a"}
Jul 13 21:09:36.944549 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.944527    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j7r2w" event={"ID":"a07bee79-0d6e-439f-b920-be4cd37930b3","Type":"ContainerStarted","Data":"a5178763238dd20d859e3c1e8dafc2de32dbaf84edf6fe82939da363285c741d"}
Jul 13 21:09:36.946372 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.946347    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcjdj" event={"ID":"f76f5b2b-3323-4299-9eef-aa9bc1763402","Type":"ContainerStarted","Data":"ab8ed69a8eeda2836fe97bfc3abfa916f58f0ff984f4a5a82d44f0cc924be243"}
Jul 13 21:09:36.947840 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.947816    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5" event={"ID":"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9","Type":"ContainerStarted","Data":"eeecdd120b636db24086f6950d707dd930a82f2a8d186c2cff0f96b3cced73eb"}
Jul 13 21:09:36.950436 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.950411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4zzn" event={"ID":"30ac7b1f-d387-4c5a-9b4a-3c99304f8d45","Type":"ContainerStarted","Data":"831510ba19b06a7b8aab68e4977d07593281a9004b5f5f1c7e9cc64ca0699f14"}
Jul 13 21:09:36.952737 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.952716    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-78cbw" event={"ID":"5233d551-7dd2-41f4-b820-99367023e3b4","Type":"ContainerStarted","Data":"645e091b1bb4a2d790990ef47a07cd3f83245535591396dbaa6c32f26853808f"}
Jul 13 21:09:36.954419 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.954397    2564 generic.go:356] "Generic (PLEG): container finished" podID="7173a77a-9b2f-4837-81d5-0863d79a9008" containerID="1dcabbef3cfed89a83c244e1b7538944dd80778dc021d954d946238706ded593" exitCode=0
Jul 13 21:09:36.954527 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.954452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwxvt" event={"ID":"7173a77a-9b2f-4837-81d5-0863d79a9008","Type":"ContainerDied","Data":"1dcabbef3cfed89a83c244e1b7538944dd80778dc021d954d946238706ded593"}
Jul 13 21:09:36.955796 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.955781    2564 generic.go:356] "Generic (PLEG): container finished" podID="8bd17d048e9092b3fa303749ed84aa38" containerID="83306b72e93f3fa46cc25a0550b42dad38759a57e66d58133cd3709ba24fb710" exitCode=0
Jul 13 21:09:36.955853 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.955838    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal" event={"ID":"8bd17d048e9092b3fa303749ed84aa38","Type":"ContainerDied","Data":"83306b72e93f3fa46cc25a0550b42dad38759a57e66d58133cd3709ba24fb710"}
Jul 13 21:09:36.957524 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.957477    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d4sdl" podStartSLOduration=2.329822999 podStartE2EDuration="22.957446457s" totalImagesPullingTime="20.627623458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:36.957157047 +0000 UTC m=+23.614814055" watchObservedRunningTime="2026-07-13 21:09:36.957446457 +0000 UTC m=+23.615103442"
Jul 13 21:09:36.975273 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:36.975239    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-78cbw" podStartSLOduration=2.341327626 podStartE2EDuration="22.97522608s" totalImagesPullingTime="20.633898454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:36.975227225 +0000 UTC m=+23.632884231" watchObservedRunningTime="2026-07-13 21:09:36.97522608 +0000 UTC m=+23.632883086"
Jul 13 21:09:37.010382 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.010347    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kcjdj" podStartSLOduration=3.354719613 podStartE2EDuration="24.010330656s" totalImagesPullingTime="20.655611043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:37.009711081 +0000 UTC m=+23.667368087" watchObservedRunningTime="2026-07-13 21:09:37.010330656 +0000 UTC m=+23.667987664"
Jul 13 21:09:37.045038 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.044996    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s4zzn" podStartSLOduration=2.420896178 podStartE2EDuration="23.044982554s" totalImagesPullingTime="20.624086376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:37.044741964 +0000 UTC m=+23.702398970" watchObservedRunningTime="2026-07-13 21:09:37.044982554 +0000 UTC m=+23.702639559"
Jul 13 21:09:37.045455 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.045431    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j7r2w" podStartSLOduration=2.368898965 podStartE2EDuration="23.045424577s" totalImagesPullingTime="20.676525612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:37.030962067 +0000 UTC m=+23.688619074" watchObservedRunningTime="2026-07-13 21:09:37.045424577 +0000 UTC m=+23.703081583"
Jul 13 21:09:37.060673 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.060605    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-152.ec2.internal" podStartSLOduration=23.060568366 podStartE2EDuration="23.060568366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:37.059971455 +0000 UTC m=+23.717628464" watchObservedRunningTime="2026-07-13 21:09:37.060568366 +0000 UTC m=+23.718225373"
Jul 13 21:09:37.816048 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.816020    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:37.816207 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:37.816118    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:37.931621 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.931595    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 21:09:37.959968 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.959933    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wj8pg" event={"ID":"40ff0e81-61a7-4cd6-8f39-d973f557e974","Type":"ContainerStarted","Data":"58a9a90ad2097aa2552a0c5aeb0145e1080c3c3a5f464e49f0a215c616904a71"}
Jul 13 21:09:37.962023 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.961997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5" event={"ID":"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9","Type":"ContainerStarted","Data":"be59529d942b3fe68a5e11bd6c9214c7608b676c9e90fe992f9aa8fa60ebb57b"}
Jul 13 21:09:37.964161 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.964138    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal" event={"ID":"8bd17d048e9092b3fa303749ed84aa38","Type":"ContainerStarted","Data":"e4e45213b1d0ab143d429780fe1f02e7911792e1213ab8337cc168af786801b4"}
Jul 13 21:09:37.974133 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:37.974088    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wj8pg" podStartSLOduration=3.397783956 podStartE2EDuration="23.974070307s" totalImagesPullingTime="20.576286351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:37.973751484 +0000 UTC m=+24.631408489" watchObservedRunningTime="2026-07-13 21:09:37.974070307 +0000 UTC m=+24.631727314"
Jul 13 21:09:38.815885 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:38.815845    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:38.815885 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:38.815861    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:38.816204 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:38.815976    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:38.816204 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:38.816105    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:38.821546 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:38.821473    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T21:09:37.931617735Z","UUID":"bd19999a-65b0-4644-b949-851ca22ef590","Handler":null,"Name":"","Endpoint":""}
Jul 13 21:09:38.823164 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:38.823144    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 21:09:38.823279 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:38.823182    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 21:09:39.816681 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.816501    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:39.817221 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:39.816787    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:39.853435 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.853412    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:39.854268 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.854248    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:39.873323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.873275    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-152.ec2.internal" podStartSLOduration=25.87325636 podStartE2EDuration="25.87325636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:37.989835553 +0000 UTC m=+24.647492558" watchObservedRunningTime="2026-07-13 21:09:39.87325636 +0000 UTC m=+26.530913370"
Jul 13 21:09:39.971809 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.971770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5" event={"ID":"5e5e8d7f-b0d8-4709-b439-9f08f86fb0e9","Type":"ContainerStarted","Data":"d07591ef321c96c3607fde59437303436e6ea7a1f670b1363a564cfd230e0c09"}
Jul 13 21:09:39.975507 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.975487    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:09:39.975940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.975918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerStarted","Data":"65a1621600c7271b56e180ca26d6c96e522da7a5fae5db43ee0eef7fe6daa311"}
Jul 13 21:09:39.976176 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.976139    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:39.976598 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.976564    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d4sdl"
Jul 13 21:09:39.988253 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:39.988217    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6t5" podStartSLOduration=3.8594593440000002 podStartE2EDuration="26.988203876s" totalImagesPullingTime="23.128744532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:39.987892088 +0000 UTC m=+26.645549093" watchObservedRunningTime="2026-07-13 21:09:39.988203876 +0000 UTC m=+26.645860882"
Jul 13 21:09:40.815995 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:40.815887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:40.815995 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:40.815991    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:40.816230 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:40.816021    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:40.816230 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:40.816096    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:41.816830 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.816660    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:41.817344 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:41.816908    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:41.982578 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.982554    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:09:41.982984 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.982962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerStarted","Data":"cc02b845575c80d6387d199685d5cf392e34ceb92e8bec2764ad4102d2a2d889"}
Jul 13 21:09:41.983356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.983192    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:41.983356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.983262    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:41.983356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.983308    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:41.983356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.983332    2564 scope.go:117] "RemoveContainer" containerID="1f0efe8e474c65a78382a913dfb21ece61f602a58e85686052c13f0a8e231c30"
Jul 13 21:09:41.985528 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.985497    2564 generic.go:356] "Generic (PLEG): container finished" podID="7173a77a-9b2f-4837-81d5-0863d79a9008" containerID="283b23dc1ac182ac6bc040b4a7f62b1beef17951e5dc66ec2573353fbe9054b2" exitCode=0
Jul 13 21:09:41.985657 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.985554    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwxvt" event={"ID":"7173a77a-9b2f-4837-81d5-0863d79a9008","Type":"ContainerDied","Data":"283b23dc1ac182ac6bc040b4a7f62b1beef17951e5dc66ec2573353fbe9054b2"}
Jul 13 21:09:41.999510 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:41.999494    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:42.001457 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:42.001430    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:09:42.816297 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:42.816268    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:42.816494 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:42.816268    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:42.816494 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:42.816395    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:42.816494 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:42.816453    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:42.991527 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:42.991501    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:09:42.991947 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:42.991911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" event={"ID":"ff20edcf-5053-44fb-af08-bc34f7a74a49","Type":"ContainerStarted","Data":"0b515e8bcb7a22beb4786d11290b3e2e81229b603259140cb103f5c870997b14"}
Jul 13 21:09:43.046211 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:43.046168    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw" podStartSLOduration=8.029784475 podStartE2EDuration="29.046147851s" totalImagesPullingTime="21.016363376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:43.045923747 +0000 UTC m=+29.703580755" watchObservedRunningTime="2026-07-13 21:09:43.046147851 +0000 UTC m=+29.703804858"
Jul 13 21:09:43.277228 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:43.277201    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fwj6w"]
Jul 13 21:09:43.277341 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:43.277314    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:43.277461 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:43.277441    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:43.281530 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:43.281506    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xl4b6"]
Jul 13 21:09:43.281691 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:43.281620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:43.281781 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:43.281729    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:43.282200 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:43.282176    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jrfk"]
Jul 13 21:09:43.282278 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:43.282263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:43.282349 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:43.282334    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:43.995939 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:43.995903    2564 generic.go:356] "Generic (PLEG): container finished" podID="7173a77a-9b2f-4837-81d5-0863d79a9008" containerID="bb4c4e744d59893df9d7cec4b77392f7e6679f945e2c93ef72789c8b185a9d68" exitCode=0
Jul 13 21:09:43.996464 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:43.995964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwxvt" event={"ID":"7173a77a-9b2f-4837-81d5-0863d79a9008","Type":"ContainerDied","Data":"bb4c4e744d59893df9d7cec4b77392f7e6679f945e2c93ef72789c8b185a9d68"}
Jul 13 21:09:44.816053 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:44.816023    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:44.816212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:44.816023    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:44.816212 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:44.816129    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:44.816212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:44.816023    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:44.816212 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:44.816201    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:44.816338 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:44.816254    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:46.002014 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:46.001852    2564 generic.go:356] "Generic (PLEG): container finished" podID="7173a77a-9b2f-4837-81d5-0863d79a9008" containerID="c0c8f5f38519036d5f95705980732ace27f8d3437ba663aa2a0c176325cace7d" exitCode=0
Jul 13 21:09:46.002414 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:46.001940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwxvt" event={"ID":"7173a77a-9b2f-4837-81d5-0863d79a9008","Type":"ContainerDied","Data":"c0c8f5f38519036d5f95705980732ace27f8d3437ba663aa2a0c176325cace7d"}
Jul 13 21:09:46.596054 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:46.596017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:46.596232 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:46.596075    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:46.596232 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.596168    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:46.596232 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.596180    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:46.596232 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.596218    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs podName:6e598283-d5e4-489e-a446-3a4fca30fb56 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:18.596202347 +0000 UTC m=+65.253859348 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs") pod "network-metrics-daemon-fwj6w" (UID: "6e598283-d5e4-489e-a446-3a4fca30fb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:09:46.596232 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.596231    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret podName:166204bf-e576-42fa-80df-02d81b311427 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:18.596225139 +0000 UTC m=+65.253882123 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret") pod "global-pull-secret-syncer-xl4b6" (UID: "166204bf-e576-42fa-80df-02d81b311427") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:09:46.696495 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:46.696465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:46.696658 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.696639    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:09:46.696731 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.696669    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:09:46.696731 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.696678    2564 projected.go:196] Error preparing data for projected volume kube-api-access-jtnnq for pod openshift-network-diagnostics/network-check-target-8jrfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:46.696840 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.696742    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq podName:a2a289c2-21cf-445d-897e-9c004b7b0ba4 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:18.696724115 +0000 UTC m=+65.354381118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jtnnq" (UniqueName: "kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq") pod "network-check-target-8jrfk" (UID: "a2a289c2-21cf-445d-897e-9c004b7b0ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:09:46.815890 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:46.815853    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:46.816034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:46.815853    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:46.816034 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.815956    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl4b6" podUID="166204bf-e576-42fa-80df-02d81b311427"
Jul 13 21:09:46.816034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:46.815977    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:46.816198 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.816039    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jrfk" podUID="a2a289c2-21cf-445d-897e-9c004b7b0ba4"
Jul 13 21:09:46.816198 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:46.816127    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwj6w" podUID="6e598283-d5e4-489e-a446-3a4fca30fb56"
Jul 13 21:09:48.608573 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.608544    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-152.ec2.internal" event="NodeReady"
Jul 13 21:09:48.609032 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.608705    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 21:09:48.640932 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.640910    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"]
Jul 13 21:09:48.643055 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.643037    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.646707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.645531    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"]
Jul 13 21:09:48.646707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.645652    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 13 21:09:48.646707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.645941    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 13 21:09:48.646707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.646138    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sql78\""
Jul 13 21:09:48.646707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.646326    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 13 21:09:48.646707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.646525    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 13 21:09:48.647320 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.647303    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.649607 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.649529    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 13 21:09:48.649854 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.649837    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2l9mp\""
Jul 13 21:09:48.649951 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.649879    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 13 21:09:48.650006 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.649843    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 13 21:09:48.654932 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.654912    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 13 21:09:48.661973 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.661943    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"]
Jul 13 21:09:48.663178 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.663161    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-wc5th"]
Jul 13 21:09:48.664909 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.664890    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m8nwq"]
Jul 13 21:09:48.665040 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.665023    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:48.666429 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.666410    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7"]
Jul 13 21:09:48.667262 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.667184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8nwq"
Jul 13 21:09:48.667617 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.667568    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 13 21:09:48.667704 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.667631    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 13 21:09:48.667884 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.667848    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t7s7s\""
Jul 13 21:09:48.668031 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.668015    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"]
Jul 13 21:09:48.668093 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.668084    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7"
Jul 13 21:09:48.669785 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.669767    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rg67s\""
Jul 13 21:09:48.670025 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.670010    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 21:09:48.670136 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.670046    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s2fq8"]
Jul 13 21:09:48.670397 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.670334    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 21:09:48.670641 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.670627    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 21:09:48.670870 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.670855    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 21:09:48.671102 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.671089    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vzkhf\""
Jul 13 21:09:48.671215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.671186    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 21:09:48.671817 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.671800    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.673983 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.673964    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 21:09:48.674254 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.674238    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gqhqf\""
Jul 13 21:09:48.674333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.674292    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 21:09:48.679040 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.679021    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-wc5th"]
Jul 13 21:09:48.679151 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.679061    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8nwq"]
Jul 13 21:09:48.679151 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.679085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7"]
Jul 13 21:09:48.682728 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.682711    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s2fq8"]
Jul 13 21:09:48.711039 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.711017    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9dfd688a-ca7d-4474-8d60-5e0760699a39-serving-cert\") pod \"service-ca-operator-7695fc6c8c-bxdrx\" (UID: \"9dfd688a-ca7d-4474-8d60-5e0760699a39\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.711145 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.711059    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9dfd688a-ca7d-4474-8d60-5e0760699a39-config\") pod \"service-ca-operator-7695fc6c8c-bxdrx\" (UID: \"9dfd688a-ca7d-4474-8d60-5e0760699a39\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.711145 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.711086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pjt2\" (UniqueName: \"kubernetes.io/projected/9dfd688a-ca7d-4474-8d60-5e0760699a39-kube-api-access-4pjt2\") pod \"service-ca-operator-7695fc6c8c-bxdrx\" (UID: \"9dfd688a-ca7d-4474-8d60-5e0760699a39\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.811960 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.811874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjt2\" (UniqueName: \"kubernetes.io/projected/9dfd688a-ca7d-4474-8d60-5e0760699a39-kube-api-access-4pjt2\") pod \"service-ca-operator-7695fc6c8c-bxdrx\" (UID: \"9dfd688a-ca7d-4474-8d60-5e0760699a39\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.811960 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.811915    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-registry-certificates\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.812172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.811967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-trusted-ca\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.812172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812001    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-tmp-dir\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.812172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e66718-f6ca-466b-80f6-ee99fc06444f-cert\") pod \"ingress-canary-m8nwq\" (UID: \"a1e66718-f6ca-466b-80f6-ee99fc06444f\") " pod="openshift-ingress-canary/ingress-canary-m8nwq"
Jul 13 21:09:48.812172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-image-registry-private-configuration\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.812172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6rvb\" (UniqueName: \"kubernetes.io/projected/b31bddbe-a6ee-4b7b-a811-50f88d6438c9-kube-api-access-l6rvb\") pod \"network-check-source-869dcfcdcd-qpxz7\" (UID: \"b31bddbe-a6ee-4b7b-a811-50f88d6438c9\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7"
Jul 13 21:09:48.812172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-config-volume\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.812172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812112    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.812172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812139    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e278d43d-f638-4047-96ca-a5ef535138d7-nginx-conf\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:48.812458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9dfd688a-ca7d-4474-8d60-5e0760699a39-serving-cert\") pod \"service-ca-operator-7695fc6c8c-bxdrx\" (UID: \"9dfd688a-ca7d-4474-8d60-5e0760699a39\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.812458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-installation-pull-secrets\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.812458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812263    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnn9\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-kube-api-access-9qnn9\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.812458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812288    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvgz\" (UniqueName: \"kubernetes.io/projected/a1e66718-f6ca-466b-80f6-ee99fc06444f-kube-api-access-hcvgz\") pod \"ingress-canary-m8nwq\" (UID: \"a1e66718-f6ca-466b-80f6-ee99fc06444f\") " pod="openshift-ingress-canary/ingress-canary-m8nwq"
Jul 13 21:09:48.812458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9dfd688a-ca7d-4474-8d60-5e0760699a39-config\") pod \"service-ca-operator-7695fc6c8c-bxdrx\" (UID: \"9dfd688a-ca7d-4474-8d60-5e0760699a39\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.812458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812401    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2281219e-2726-490f-80cb-cb963347c09e-ca-trust-extracted\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.812458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812429    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-bound-sa-token\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.812762 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.812762 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812509    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:48.812762 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzn25\" (UniqueName: \"kubernetes.io/projected/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-kube-api-access-jzn25\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.812894 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.812867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9dfd688a-ca7d-4474-8d60-5e0760699a39-config\") pod \"service-ca-operator-7695fc6c8c-bxdrx\" (UID: \"9dfd688a-ca7d-4474-8d60-5e0760699a39\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.816077 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.816057    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:09:48.816159 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.816149    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:09:48.816292 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.816266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:09:48.816537 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.816514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9dfd688a-ca7d-4474-8d60-5e0760699a39-serving-cert\") pod \"service-ca-operator-7695fc6c8c-bxdrx\" (UID: \"9dfd688a-ca7d-4474-8d60-5e0760699a39\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.818379 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.818360    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 21:09:48.818886 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.818561    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 21:09:48.818886 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.818676    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fgl7x\""
Jul 13 21:09:48.818886 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.818698    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gq4dz\""
Jul 13 21:09:48.820616 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.820576    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjt2\" (UniqueName: \"kubernetes.io/projected/9dfd688a-ca7d-4474-8d60-5e0760699a39-kube-api-access-4pjt2\") pod \"service-ca-operator-7695fc6c8c-bxdrx\" (UID: \"9dfd688a-ca7d-4474-8d60-5e0760699a39\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.913185 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913163    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rvb\" (UniqueName: \"kubernetes.io/projected/b31bddbe-a6ee-4b7b-a811-50f88d6438c9-kube-api-access-l6rvb\") pod \"network-check-source-869dcfcdcd-qpxz7\" (UID: \"b31bddbe-a6ee-4b7b-a811-50f88d6438c9\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7"
Jul 13 21:09:48.913334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913196    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-config-volume\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.913334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.913334 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:48.913316    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:09:48.913334 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:48.913325    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f: secret "image-registry-tls" not found
Jul 13 21:09:48.913532 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:48.913378    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls podName:2281219e-2726-490f-80cb-cb963347c09e nodeName:}" failed. No retries permitted until 2026-07-13 21:09:49.413357747 +0000 UTC m=+36.071014748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls") pod "image-registry-5ccd5cb9f6-z8v8f" (UID: "2281219e-2726-490f-80cb-cb963347c09e") : secret "image-registry-tls" not found
Jul 13 21:09:48.913532 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e278d43d-f638-4047-96ca-a5ef535138d7-nginx-conf\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:48.913684 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-installation-pull-secrets\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.913684 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913563    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnn9\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-kube-api-access-9qnn9\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.913684 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvgz\" (UniqueName: \"kubernetes.io/projected/a1e66718-f6ca-466b-80f6-ee99fc06444f-kube-api-access-hcvgz\") pod \"ingress-canary-m8nwq\" (UID: \"a1e66718-f6ca-466b-80f6-ee99fc06444f\") " pod="openshift-ingress-canary/ingress-canary-m8nwq"
Jul 13 21:09:48.913907 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913883    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2281219e-2726-490f-80cb-cb963347c09e-ca-trust-extracted\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.913996 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-bound-sa-token\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.913996 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913894    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-config-volume\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.913996 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.913962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.914139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:48.914139 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:48.914036    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:09:48.914139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914055    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzn25\" (UniqueName: \"kubernetes.io/projected/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-kube-api-access-jzn25\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.914139 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:48.914078    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls podName:fd368277-a7bf-4c9e-bdcc-90aea0f41dbb nodeName:}" failed. No retries permitted until 2026-07-13 21:09:49.414066538 +0000 UTC m=+36.071723522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls") pod "dns-default-s2fq8" (UID: "fd368277-a7bf-4c9e-bdcc-90aea0f41dbb") : secret "dns-default-metrics-tls" not found
Jul 13 21:09:48.914139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-registry-certificates\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.914379 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914154    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-trusted-ca\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.914379 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2281219e-2726-490f-80cb-cb963347c09e-ca-trust-extracted\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.914379 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914231    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-tmp-dir\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.914379 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e66718-f6ca-466b-80f6-ee99fc06444f-cert\") pod \"ingress-canary-m8nwq\" (UID: \"a1e66718-f6ca-466b-80f6-ee99fc06444f\") " pod="openshift-ingress-canary/ingress-canary-m8nwq"
Jul 13 21:09:48.914379 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-image-registry-private-configuration\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.914657 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:48.914235    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 21:09:48.914657 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:48.914506    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert podName:e278d43d-f638-4047-96ca-a5ef535138d7 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:49.414486777 +0000 UTC m=+36.072143761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wc5th" (UID: "e278d43d-f638-4047-96ca-a5ef535138d7") : secret "networking-console-plugin-cert" not found
Jul 13 21:09:48.914657 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-registry-certificates\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.914657 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e278d43d-f638-4047-96ca-a5ef535138d7-nginx-conf\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:48.914860 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.914778    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-tmp-dir\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.915227 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.915209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-trusted-ca\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.916259 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.916241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-installation-pull-secrets\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.917252 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.917220    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e66718-f6ca-466b-80f6-ee99fc06444f-cert\") pod \"ingress-canary-m8nwq\" (UID: \"a1e66718-f6ca-466b-80f6-ee99fc06444f\") " pod="openshift-ingress-canary/ingress-canary-m8nwq"
Jul 13 21:09:48.919548 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.919529    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-image-registry-private-configuration\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.922172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.922130    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvgz\" (UniqueName: \"kubernetes.io/projected/a1e66718-f6ca-466b-80f6-ee99fc06444f-kube-api-access-hcvgz\") pod \"ingress-canary-m8nwq\" (UID: \"a1e66718-f6ca-466b-80f6-ee99fc06444f\") " pod="openshift-ingress-canary/ingress-canary-m8nwq"
Jul 13 21:09:48.922679 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.922657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6rvb\" (UniqueName: \"kubernetes.io/projected/b31bddbe-a6ee-4b7b-a811-50f88d6438c9-kube-api-access-l6rvb\") pod \"network-check-source-869dcfcdcd-qpxz7\" (UID: \"b31bddbe-a6ee-4b7b-a811-50f88d6438c9\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7"
Jul 13 21:09:48.923035 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.923006    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnn9\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-kube-api-access-9qnn9\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.923140 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.923041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-bound-sa-token\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:48.923311 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.923225    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzn25\" (UniqueName: \"kubernetes.io/projected/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-kube-api-access-jzn25\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:48.955819 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.955788    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"
Jul 13 21:09:48.988008 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.987977    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8nwq"
Jul 13 21:09:48.989429 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.989398    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:09:48.996885 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:48.996865    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7"
Jul 13 21:09:49.073028 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:49.072976    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:09:49.418766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:49.418681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:49.418766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:49.418735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:49.418949 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:49.418833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:49.418949 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:49.418845    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:09:49.418949 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:49.418882    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 21:09:49.418949 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:49.418910    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls podName:fd368277-a7bf-4c9e-bdcc-90aea0f41dbb nodeName:}" failed. No retries permitted until 2026-07-13 21:09:50.418890855 +0000 UTC m=+37.076547856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls") pod "dns-default-s2fq8" (UID: "fd368277-a7bf-4c9e-bdcc-90aea0f41dbb") : secret "dns-default-metrics-tls" not found
Jul 13 21:09:49.418949 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:49.418926    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:09:49.418949 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:49.418936    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f: secret "image-registry-tls" not found
Jul 13 21:09:49.419195 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:49.418938    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert podName:e278d43d-f638-4047-96ca-a5ef535138d7 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:50.418925701 +0000 UTC m=+37.076582685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wc5th" (UID: "e278d43d-f638-4047-96ca-a5ef535138d7") : secret "networking-console-plugin-cert" not found
Jul 13 21:09:49.419195 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:49.419007    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls podName:2281219e-2726-490f-80cb-cb963347c09e nodeName:}" failed. No retries permitted until 2026-07-13 21:09:50.418986566 +0000 UTC m=+37.076643555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls") pod "image-registry-5ccd5cb9f6-z8v8f" (UID: "2281219e-2726-490f-80cb-cb963347c09e") : secret "image-registry-tls" not found
Jul 13 21:09:50.262335 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:50.262114    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8nwq"]
Jul 13 21:09:50.267777 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:50.267749    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx"]
Jul 13 21:09:50.272407 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:50.272385    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7"]
Jul 13 21:09:50.276865 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:50.276833    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31bddbe_a6ee_4b7b_a811_50f88d6438c9.slice/crio-701a8d648f7107bed52d15b49643e8e49aea82bcd2096d0566c686c66fcc3447 WatchSource:0}: Error finding container 701a8d648f7107bed52d15b49643e8e49aea82bcd2096d0566c686c66fcc3447: Status 404 returned error can't find the container with id 701a8d648f7107bed52d15b49643e8e49aea82bcd2096d0566c686c66fcc3447
Jul 13 21:09:50.277868 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:50.277842    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e66718_f6ca_466b_80f6_ee99fc06444f.slice/crio-230ed492693e1eb08bde97371e48c558b3369ba4a501660d28c24ce6f18d4e4b WatchSource:0}: Error finding container 230ed492693e1eb08bde97371e48c558b3369ba4a501660d28c24ce6f18d4e4b: Status 404 returned error can't find the container with id 230ed492693e1eb08bde97371e48c558b3369ba4a501660d28c24ce6f18d4e4b
Jul 13 21:09:50.278713 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:50.278651    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfd688a_ca7d_4474_8d60_5e0760699a39.slice/crio-c8086d8a303e7d7aaaa0f9d0a2bece6afd4d3f24b26016126b49f1c5550e4d1a WatchSource:0}: Error finding container c8086d8a303e7d7aaaa0f9d0a2bece6afd4d3f24b26016126b49f1c5550e4d1a: Status 404 returned error can't find the container with id c8086d8a303e7d7aaaa0f9d0a2bece6afd4d3f24b26016126b49f1c5550e4d1a
Jul 13 21:09:50.426797 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:50.426714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:50.426797 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:50.426766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:50.426797 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:50.426785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:50.426993 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:50.426880    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 21:09:50.426993 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:50.426880    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:09:50.426993 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:50.426888    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:09:50.426993 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:50.426901    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f: secret "image-registry-tls" not found
Jul 13 21:09:50.426993 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:50.426932    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert podName:e278d43d-f638-4047-96ca-a5ef535138d7 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:52.426918704 +0000 UTC m=+39.084575688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wc5th" (UID: "e278d43d-f638-4047-96ca-a5ef535138d7") : secret "networking-console-plugin-cert" not found
Jul 13 21:09:50.426993 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:50.426956    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls podName:fd368277-a7bf-4c9e-bdcc-90aea0f41dbb nodeName:}" failed. No retries permitted until 2026-07-13 21:09:52.426938081 +0000 UTC m=+39.084595115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls") pod "dns-default-s2fq8" (UID: "fd368277-a7bf-4c9e-bdcc-90aea0f41dbb") : secret "dns-default-metrics-tls" not found
Jul 13 21:09:50.426993 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:50.426977    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls podName:2281219e-2726-490f-80cb-cb963347c09e nodeName:}" failed. No retries permitted until 2026-07-13 21:09:52.426966825 +0000 UTC m=+39.084623815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls") pod "image-registry-5ccd5cb9f6-z8v8f" (UID: "2281219e-2726-490f-80cb-cb963347c09e") : secret "image-registry-tls" not found
Jul 13 21:09:51.000431 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.000397    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"]
Jul 13 21:09:51.002208 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.002183    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"
Jul 13 21:09:51.004106 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.004085    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"]
Jul 13 21:09:51.004522 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.004502    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bwwqb\""
Jul 13 21:09:51.005120 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.005099    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 13 21:09:51.005196 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.005117    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 13 21:09:51.005862 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.005843    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 13 21:09:51.009203 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.007209    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 13 21:09:51.011103 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.010271    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"]
Jul 13 21:09:51.011103 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.010574    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.012292 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.012272    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.015394 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.015372    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx" event={"ID":"9dfd688a-ca7d-4474-8d60-5e0760699a39","Type":"ContainerStarted","Data":"c8086d8a303e7d7aaaa0f9d0a2bece6afd4d3f24b26016126b49f1c5550e4d1a"}
Jul 13 21:09:51.017083 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.017060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8nwq" event={"ID":"a1e66718-f6ca-466b-80f6-ee99fc06444f","Type":"ContainerStarted","Data":"230ed492693e1eb08bde97371e48c558b3369ba4a501660d28c24ce6f18d4e4b"}
Jul 13 21:09:51.020721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.020690    2564 generic.go:356] "Generic (PLEG): container finished" podID="7173a77a-9b2f-4837-81d5-0863d79a9008" containerID="83d870d4cb73b6a37167af1868d10ba3395f2382a18565f5d8f695ade1baa252" exitCode=0
Jul 13 21:09:51.020820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.020756    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwxvt" event={"ID":"7173a77a-9b2f-4837-81d5-0863d79a9008","Type":"ContainerDied","Data":"83d870d4cb73b6a37167af1868d10ba3395f2382a18565f5d8f695ade1baa252"}
Jul 13 21:09:51.022384 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.022361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7" event={"ID":"b31bddbe-a6ee-4b7b-a811-50f88d6438c9","Type":"ContainerStarted","Data":"701a8d648f7107bed52d15b49643e8e49aea82bcd2096d0566c686c66fcc3447"}
Jul 13 21:09:51.026752 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.026395    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"]
Jul 13 21:09:51.028710 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.027633    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"]
Jul 13 21:09:51.028710 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.027889    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 13 21:09:51.028710 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.028075    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 13 21:09:51.029318 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.028969    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 13 21:09:51.029318 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.029172    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 13 21:09:51.029511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.029493    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"]
Jul 13 21:09:51.030516 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.030497    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fc53cd46-1d12-43bf-8076-0360e06cbb3b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6df7c9f997-z7b8j\" (UID: \"fc53cd46-1d12-43bf-8076-0360e06cbb3b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130512    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4mt\" (UniqueName: \"kubernetes.io/projected/858ce0cc-f7ad-4532-a2e1-dea2ccced699-kube-api-access-jk4mt\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-ca\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130595    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-hub\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c5cd1ca-582e-400f-9829-fb03d2f7b654-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5846ddbf56-kr5kp\" (UID: \"1c5cd1ca-582e-400f-9829-fb03d2f7b654\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5w7\" (UniqueName: \"kubernetes.io/projected/1c5cd1ca-582e-400f-9829-fb03d2f7b654-kube-api-access-bj5w7\") pod \"managed-serviceaccount-addon-agent-5846ddbf56-kr5kp\" (UID: \"1c5cd1ca-582e-400f-9829-fb03d2f7b654\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/858ce0cc-f7ad-4532-a2e1-dea2ccced699-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.130940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.130920    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc53cd46-1d12-43bf-8076-0360e06cbb3b-tmp\") pod \"klusterlet-addon-workmgr-6df7c9f997-z7b8j\" (UID: \"fc53cd46-1d12-43bf-8076-0360e06cbb3b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.131415 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.131232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjh9l\" (UniqueName: \"kubernetes.io/projected/fc53cd46-1d12-43bf-8076-0360e06cbb3b-kube-api-access-vjh9l\") pod \"klusterlet-addon-workmgr-6df7c9f997-z7b8j\" (UID: \"fc53cd46-1d12-43bf-8076-0360e06cbb3b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.232565 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/858ce0cc-f7ad-4532-a2e1-dea2ccced699-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.232721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc53cd46-1d12-43bf-8076-0360e06cbb3b-tmp\") pod \"klusterlet-addon-workmgr-6df7c9f997-z7b8j\" (UID: \"fc53cd46-1d12-43bf-8076-0360e06cbb3b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.232721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjh9l\" (UniqueName: \"kubernetes.io/projected/fc53cd46-1d12-43bf-8076-0360e06cbb3b-kube-api-access-vjh9l\") pod \"klusterlet-addon-workmgr-6df7c9f997-z7b8j\" (UID: \"fc53cd46-1d12-43bf-8076-0360e06cbb3b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.232721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fc53cd46-1d12-43bf-8076-0360e06cbb3b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6df7c9f997-z7b8j\" (UID: \"fc53cd46-1d12-43bf-8076-0360e06cbb3b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.232895 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.232895 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4mt\" (UniqueName: \"kubernetes.io/projected/858ce0cc-f7ad-4532-a2e1-dea2ccced699-kube-api-access-jk4mt\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.232895 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-ca\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.232895 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-hub\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.232895 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c5cd1ca-582e-400f-9829-fb03d2f7b654-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5846ddbf56-kr5kp\" (UID: \"1c5cd1ca-582e-400f-9829-fb03d2f7b654\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"
Jul 13 21:09:51.232895 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232888    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5w7\" (UniqueName: \"kubernetes.io/projected/1c5cd1ca-582e-400f-9829-fb03d2f7b654-kube-api-access-bj5w7\") pod \"managed-serviceaccount-addon-agent-5846ddbf56-kr5kp\" (UID: \"1c5cd1ca-582e-400f-9829-fb03d2f7b654\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"
Jul 13 21:09:51.233178 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.232938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.233883 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.233290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/858ce0cc-f7ad-4532-a2e1-dea2ccced699-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.233883 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.233636    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc53cd46-1d12-43bf-8076-0360e06cbb3b-tmp\") pod \"klusterlet-addon-workmgr-6df7c9f997-z7b8j\" (UID: \"fc53cd46-1d12-43bf-8076-0360e06cbb3b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.238785 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.238112    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-ca\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.238785 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.238192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-hub\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.238785 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.238653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.238785 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.238736    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/858ce0cc-f7ad-4532-a2e1-dea2ccced699-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.240559 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.240513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c5cd1ca-582e-400f-9829-fb03d2f7b654-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5846ddbf56-kr5kp\" (UID: \"1c5cd1ca-582e-400f-9829-fb03d2f7b654\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"
Jul 13 21:09:51.241255 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.241218    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fc53cd46-1d12-43bf-8076-0360e06cbb3b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6df7c9f997-z7b8j\" (UID: \"fc53cd46-1d12-43bf-8076-0360e06cbb3b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.242083 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.242041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5w7\" (UniqueName: \"kubernetes.io/projected/1c5cd1ca-582e-400f-9829-fb03d2f7b654-kube-api-access-bj5w7\") pod \"managed-serviceaccount-addon-agent-5846ddbf56-kr5kp\" (UID: \"1c5cd1ca-582e-400f-9829-fb03d2f7b654\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"
Jul 13 21:09:51.243027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.243004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4mt\" (UniqueName: \"kubernetes.io/projected/858ce0cc-f7ad-4532-a2e1-dea2ccced699-kube-api-access-jk4mt\") pod \"cluster-proxy-proxy-agent-6975fb5dc7-bpdgs\" (UID: \"858ce0cc-f7ad-4532-a2e1-dea2ccced699\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.243563 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.243538    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjh9l\" (UniqueName: \"kubernetes.io/projected/fc53cd46-1d12-43bf-8076-0360e06cbb3b-kube-api-access-vjh9l\") pod \"klusterlet-addon-workmgr-6df7c9f997-z7b8j\" (UID: \"fc53cd46-1d12-43bf-8076-0360e06cbb3b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.324472 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.324407    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"
Jul 13 21:09:51.339698 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.339675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:09:51.357108 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.356869    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:09:51.518431 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.518359    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp"]
Jul 13 21:09:51.526730 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:51.526701    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c5cd1ca_582e_400f_9829_fb03d2f7b654.slice/crio-c66505ce0fa82d204664b01a58b1195786b89f5f106b3b9a76246fb620dca140 WatchSource:0}: Error finding container c66505ce0fa82d204664b01a58b1195786b89f5f106b3b9a76246fb620dca140: Status 404 returned error can't find the container with id c66505ce0fa82d204664b01a58b1195786b89f5f106b3b9a76246fb620dca140
Jul 13 21:09:51.535931 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.535876    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"]
Jul 13 21:09:51.539939 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:09:51.539912    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod858ce0cc_f7ad_4532_a2e1_dea2ccced699.slice/crio-24104fffa2e07354c7472b22bbc23482bd8efe23e63ca4f2a087b8d439019a79 WatchSource:0}: Error finding container 24104fffa2e07354c7472b22bbc23482bd8efe23e63ca4f2a087b8d439019a79: Status 404 returned error can't find the container with id 24104fffa2e07354c7472b22bbc23482bd8efe23e63ca4f2a087b8d439019a79
Jul 13 21:09:51.546362 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:51.546342    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"]
Jul 13 21:09:52.036659 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:52.036563    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" event={"ID":"858ce0cc-f7ad-4532-a2e1-dea2ccced699","Type":"ContainerStarted","Data":"24104fffa2e07354c7472b22bbc23482bd8efe23e63ca4f2a087b8d439019a79"}
Jul 13 21:09:52.038018 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:52.037990    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp" event={"ID":"1c5cd1ca-582e-400f-9829-fb03d2f7b654","Type":"ContainerStarted","Data":"c66505ce0fa82d204664b01a58b1195786b89f5f106b3b9a76246fb620dca140"}
Jul 13 21:09:52.049413 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:52.048435    2564 generic.go:356] "Generic (PLEG): container finished" podID="7173a77a-9b2f-4837-81d5-0863d79a9008" containerID="29d42888db4129a74446f4a1c0d5dcc2713c2142597ebe35a18527f0c3da9067" exitCode=0
Jul 13 21:09:52.049413 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:52.048517    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwxvt" event={"ID":"7173a77a-9b2f-4837-81d5-0863d79a9008","Type":"ContainerDied","Data":"29d42888db4129a74446f4a1c0d5dcc2713c2142597ebe35a18527f0c3da9067"}
Jul 13 21:09:52.051334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:52.051314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j" event={"ID":"fc53cd46-1d12-43bf-8076-0360e06cbb3b","Type":"ContainerStarted","Data":"901c422246feee95a323ce78090ddf74d80bbf77343d7cd3005a95fad56ddfa4"}
Jul 13 21:09:52.447238 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:52.447142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:52.447238 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:52.447219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:52.448071 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:52.447300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:52.448071 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:52.447566    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:09:52.448071 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:52.447602    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f: secret "image-registry-tls" not found
Jul 13 21:09:52.448071 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:52.447669    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls podName:2281219e-2726-490f-80cb-cb963347c09e nodeName:}" failed. No retries permitted until 2026-07-13 21:09:56.447646106 +0000 UTC m=+43.105303112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls") pod "image-registry-5ccd5cb9f6-z8v8f" (UID: "2281219e-2726-490f-80cb-cb963347c09e") : secret "image-registry-tls" not found
Jul 13 21:09:52.448234 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:52.448089    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:09:52.448234 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:52.448129    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls podName:fd368277-a7bf-4c9e-bdcc-90aea0f41dbb nodeName:}" failed. No retries permitted until 2026-07-13 21:09:56.448117002 +0000 UTC m=+43.105773985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls") pod "dns-default-s2fq8" (UID: "fd368277-a7bf-4c9e-bdcc-90aea0f41dbb") : secret "dns-default-metrics-tls" not found
Jul 13 21:09:52.448330 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:52.448234    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 21:09:52.448330 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:52.448309    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert podName:e278d43d-f638-4047-96ca-a5ef535138d7 nodeName:}" failed. No retries permitted until 2026-07-13 21:09:56.44829018 +0000 UTC m=+43.105947176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wc5th" (UID: "e278d43d-f638-4047-96ca-a5ef535138d7") : secret "networking-console-plugin-cert" not found
Jul 13 21:09:53.068222 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:53.067315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwxvt" event={"ID":"7173a77a-9b2f-4837-81d5-0863d79a9008","Type":"ContainerStarted","Data":"7d9faedd18162e2f1d72ecfa11a428bd6ea182d63853d74cc0d1a1c35bd34ffe"}
Jul 13 21:09:53.091635 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:53.091262    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xwxvt" podStartSLOduration=7.851571578 podStartE2EDuration="39.091239288s" totalImagesPullingTime="31.23966771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:09:53.088667455 +0000 UTC m=+39.746324465" watchObservedRunningTime="2026-07-13 21:09:53.091239288 +0000 UTC m=+39.748896293"
Jul 13 21:09:53.212676 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:53.212543    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:09:53.286208 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:53.286118    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:09:53.392716 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:53.392660    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:09:53.463636 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:53.463539    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:09:56.483546 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:56.483507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:09:56.484018 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:56.483600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:09:56.484018 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:56.483660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:09:56.484018 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:56.483700    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 21:09:56.484018 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:56.483761    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:09:56.484018 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:56.483766    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert podName:e278d43d-f638-4047-96ca-a5ef535138d7 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:04.483747335 +0000 UTC m=+51.141404336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wc5th" (UID: "e278d43d-f638-4047-96ca-a5ef535138d7") : secret "networking-console-plugin-cert" not found
Jul 13 21:09:56.484018 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:56.483765    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:09:56.484018 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:56.483790    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f: secret "image-registry-tls" not found
Jul 13 21:09:56.484018 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:56.483803    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls podName:fd368277-a7bf-4c9e-bdcc-90aea0f41dbb nodeName:}" failed. No retries permitted until 2026-07-13 21:10:04.48379365 +0000 UTC m=+51.141450634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls") pod "dns-default-s2fq8" (UID: "fd368277-a7bf-4c9e-bdcc-90aea0f41dbb") : secret "dns-default-metrics-tls" not found
Jul 13 21:09:56.484018 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:09:56.483846    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls podName:2281219e-2726-490f-80cb-cb963347c09e nodeName:}" failed. No retries permitted until 2026-07-13 21:10:04.483827824 +0000 UTC m=+51.141484813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls") pod "image-registry-5ccd5cb9f6-z8v8f" (UID: "2281219e-2726-490f-80cb-cb963347c09e") : secret "image-registry-tls" not found
Jul 13 21:09:57.086719 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:57.086679    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:09:57.187813 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:57.187769    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:09:58.052905 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:58.052828    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:09:58.152944 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:09:58.152908    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:00.777120 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:00.777087    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:00.877405 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:00.877337    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:01.685262 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:01.685172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:01.785085 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:01.785048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:02.098481 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.098442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8nwq" event={"ID":"a1e66718-f6ca-466b-80f6-ee99fc06444f","Type":"ContainerStarted","Data":"0fba830989fde79f3577d6aeaeda86b7ff082a6d7a980d9b9a37c4d8b0adc6a1"}
Jul 13 21:10:02.101154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.101066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" event={"ID":"858ce0cc-f7ad-4532-a2e1-dea2ccced699","Type":"ContainerStarted","Data":"c01f869ce440d12feefdc4023c545da973ad80b444bf2af35f185e86a6bb5d1e"}
Jul 13 21:10:02.101154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.101110    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" event={"ID":"858ce0cc-f7ad-4532-a2e1-dea2ccced699","Type":"ContainerStarted","Data":"2e842c9de4c175e47d5848d890f162a00c8da995e4de2a49fe80c89941875ef5"}
Jul 13 21:10:02.101154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.101129    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" event={"ID":"858ce0cc-f7ad-4532-a2e1-dea2ccced699","Type":"ContainerStarted","Data":"70f416da16399ff9627bb1c54515ad02b22db63fe853f39e91b985c582f228c7"}
Jul 13 21:10:02.102762 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.102740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp" event={"ID":"1c5cd1ca-582e-400f-9829-fb03d2f7b654","Type":"ContainerStarted","Data":"31515d8e2a3e8ebcc870a27d4f3f07e38d2995d82ef84b68f4530c7aa3f56fd1"}
Jul 13 21:10:02.104717 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.104305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx" event={"ID":"9dfd688a-ca7d-4474-8d60-5e0760699a39","Type":"ContainerStarted","Data":"1ee2db52500f1e0cebad68001f6775bc220e975af9919835606381c7e1ba233c"}
Jul 13 21:10:02.105995 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.105970    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j" event={"ID":"fc53cd46-1d12-43bf-8076-0360e06cbb3b","Type":"ContainerStarted","Data":"bce5fac5bd66f473d7ec6f9e4ef82fc65cb818eb7b34330973e0277fe214d213"}
Jul 13 21:10:02.106091 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.106019    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:10:02.108398 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.107880    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7" event={"ID":"b31bddbe-a6ee-4b7b-a811-50f88d6438c9","Type":"ContainerStarted","Data":"98bd4a821071847670bf1def56bcb64981e96c50247e18d95fa814e40e13531b"}
Jul 13 21:10:02.108398 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.108050    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j"
Jul 13 21:10:02.132910 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.132865    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m8nwq" podStartSLOduration=3.085473457 podStartE2EDuration="14.132851798s" totalImagesPullingTime="11.047378341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:02.114795318 +0000 UTC m=+48.772452326" watchObservedRunningTime="2026-07-13 21:10:02.132851798 +0000 UTC m=+48.790508803"
Jul 13 21:10:02.149698 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.149651    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx" podStartSLOduration=37.106057772 podStartE2EDuration="48.149634614s" totalImagesPullingTime="11.043576842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:02.148957422 +0000 UTC m=+48.806614429" watchObservedRunningTime="2026-07-13 21:10:02.149634614 +0000 UTC m=+48.807291622"
Jul 13 21:10:02.149851 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.149827    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qpxz7" podStartSLOduration=36.079834222 podStartE2EDuration="47.149817764s" totalImagesPullingTime="11.069983542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:02.133290676 +0000 UTC m=+48.790947681" watchObservedRunningTime="2026-07-13 21:10:02.149817764 +0000 UTC m=+48.807474771"
Jul 13 21:10:02.163420 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.163374    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846ddbf56-kr5kp" podStartSLOduration=2.336534089 podStartE2EDuration="12.16336207s" totalImagesPullingTime="9.826827981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:02.162768009 +0000 UTC m=+48.820425015" watchObservedRunningTime="2026-07-13 21:10:02.16336207 +0000 UTC m=+48.821019076"
Jul 13 21:10:02.180256 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.180222    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" podStartSLOduration=2.371262301 podStartE2EDuration="12.180211466s" totalImagesPullingTime="9.808949165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:02.178967484 +0000 UTC m=+48.836624501" watchObservedRunningTime="2026-07-13 21:10:02.180211466 +0000 UTC m=+48.837868476"
Jul 13 21:10:02.194599 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.194546    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6df7c9f997-z7b8j" podStartSLOduration=2.395200076 podStartE2EDuration="12.194529685s" totalImagesPullingTime="9.799329609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:02.193450932 +0000 UTC m=+48.851107938" watchObservedRunningTime="2026-07-13 21:10:02.194529685 +0000 UTC m=+48.852186692"
Jul 13 21:10:02.492377 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.492333    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:02.541823 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.541795    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:02.765265 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.765187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:02.866272 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.866239    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:02.970694 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:02.970653    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:03.020029 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:03.019955    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:03.210164 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:03.210127    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:03.494436 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:03.494401    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:04.552069 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:04.552026    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:10:04.552525 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:04.552096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:10:04.552525 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:04.552138    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:10:04.552525 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:04.552167    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 21:10:04.552525 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:04.552255    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert podName:e278d43d-f638-4047-96ca-a5ef535138d7 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:20.552235428 +0000 UTC m=+67.209892411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wc5th" (UID: "e278d43d-f638-4047-96ca-a5ef535138d7") : secret "networking-console-plugin-cert" not found
Jul 13 21:10:04.552525 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:04.552256    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:10:04.552525 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:04.552288    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:10:04.552525 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:04.552309    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f: secret "image-registry-tls" not found
Jul 13 21:10:04.552525 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:04.552324    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls podName:fd368277-a7bf-4c9e-bdcc-90aea0f41dbb nodeName:}" failed. No retries permitted until 2026-07-13 21:10:20.552306684 +0000 UTC m=+67.209963685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls") pod "dns-default-s2fq8" (UID: "fd368277-a7bf-4c9e-bdcc-90aea0f41dbb") : secret "dns-default-metrics-tls" not found
Jul 13 21:10:04.552525 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:04.552362    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls podName:2281219e-2726-490f-80cb-cb963347c09e nodeName:}" failed. No retries permitted until 2026-07-13 21:10:20.552347684 +0000 UTC m=+67.210004671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls") pod "image-registry-5ccd5cb9f6-z8v8f" (UID: "2281219e-2726-490f-80cb-cb963347c09e") : secret "image-registry-tls" not found
Jul 13 21:10:04.833421 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:04.833334    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:05.777715 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:05.777682    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:08.860163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:08.860119    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:09.938954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:09.938921    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:10.131559 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:10.131527    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:10.233031 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:10.232953    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:10.333192 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:10.333169    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:10.533926 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:10.533897    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:10.634460 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:10.634431    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:10.687378 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:10.687352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:11.630395 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:11.630359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:12.023362 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:12.023327    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:12.228693 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:12.228661    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:12.431717 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:12.431645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:12.630449 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:12.630417    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:13.430628 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:13.430576    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:13.628089 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:13.628063    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:13.848853 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:13.848829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:14.017685 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:14.017656    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx6cw"
Jul 13 21:10:14.026154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:14.026131    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:14.232609 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:14.232508    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:14.284331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:14.284302    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:14.426742 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:14.426714    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:14.829847 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:14.829816    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:15.226308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:15.226224    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:16.025059 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:16.025016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:16.355403 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:16.355326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:16.456638 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:16.456609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:16.633374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:16.633296    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:16.828985 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:16.828956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:17.026567 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:17.026533    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:17.293713 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:17.293623    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:17.831545 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:17.831513    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:18.265682 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.265650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:18.426945 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.426915    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:18.648415 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.648351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:10:18.648529 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.648446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:10:18.651030 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.651013    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 21:10:18.651936 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.651919    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 21:10:18.661495 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.661475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/166204bf-e576-42fa-80df-02d81b311427-original-pull-secret\") pod \"global-pull-secret-syncer-xl4b6\" (UID: \"166204bf-e576-42fa-80df-02d81b311427\") " pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:10:18.661622 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.661505    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e598283-d5e4-489e-a446-3a4fca30fb56-metrics-certs\") pod \"network-metrics-daemon-fwj6w\" (UID: \"6e598283-d5e4-489e-a446-3a4fca30fb56\") " pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:10:18.749171 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.749136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:10:18.751485 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.751468    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnnq\" (UniqueName: \"kubernetes.io/projected/a2a289c2-21cf-445d-897e-9c004b7b0ba4-kube-api-access-jtnnq\") pod \"network-check-target-8jrfk\" (UID: \"a2a289c2-21cf-445d-897e-9c004b7b0ba4\") " pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:10:18.837435 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.837412    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gq4dz\""
Jul 13 21:10:18.841395 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.841377    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl4b6"
Jul 13 21:10:18.845254 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.845230    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:10:18.848763 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.848744    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fgl7x\""
Jul 13 21:10:18.856853 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.856830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwj6w"
Jul 13 21:10:18.985381 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:18.985352    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jrfk"]
Jul 13 21:10:18.988844 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:10:18.988816    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a289c2_21cf_445d_897e_9c004b7b0ba4.slice/crio-e8c0b9db832619864ebcc3d1c81516bd7785aafd1fb03ac482011406ca0d871f WatchSource:0}: Error finding container e8c0b9db832619864ebcc3d1c81516bd7785aafd1fb03ac482011406ca0d871f: Status 404 returned error can't find the container with id e8c0b9db832619864ebcc3d1c81516bd7785aafd1fb03ac482011406ca0d871f
Jul 13 21:10:19.002360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.002299    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xl4b6"]
Jul 13 21:10:19.005438 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:10:19.005395    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166204bf_e576_42fa_80df_02d81b311427.slice/crio-93c8561be32c37e83da2e1278aea7c2e3f63f5c9807fb335bd7d014b58a12dac WatchSource:0}: Error finding container 93c8561be32c37e83da2e1278aea7c2e3f63f5c9807fb335bd7d014b58a12dac: Status 404 returned error can't find the container with id 93c8561be32c37e83da2e1278aea7c2e3f63f5c9807fb335bd7d014b58a12dac
Jul 13 21:10:19.025743 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.023153    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fwj6w"]
Jul 13 21:10:19.027619 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.027567    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:19.161046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.160989    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fwj6w" event={"ID":"6e598283-d5e4-489e-a446-3a4fca30fb56","Type":"ContainerStarted","Data":"ef55b793d0642761b52add152cac67327ac6f97170696c05a0371047dd1ed54a"}
Jul 13 21:10:19.162397 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.162372    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jrfk" event={"ID":"a2a289c2-21cf-445d-897e-9c004b7b0ba4","Type":"ContainerStarted","Data":"95bb2d26f9361e1e8b2bbf616f16711a1ab7e1ab2afe2dbd892de5b2b8bdfc3b"}
Jul 13 21:10:19.162498 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.162418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jrfk" event={"ID":"a2a289c2-21cf-445d-897e-9c004b7b0ba4","Type":"ContainerStarted","Data":"e8c0b9db832619864ebcc3d1c81516bd7785aafd1fb03ac482011406ca0d871f"}
Jul 13 21:10:19.162498 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.162442    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:10:19.163455 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.163438    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xl4b6" event={"ID":"166204bf-e576-42fa-80df-02d81b311427","Type":"ContainerStarted","Data":"93c8561be32c37e83da2e1278aea7c2e3f63f5c9807fb335bd7d014b58a12dac"}
Jul 13 21:10:19.629502 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.629467    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:19.824061 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:19.824019    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:20.027804 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.027765    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:20.168761 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.168732    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fwj6w" event={"ID":"6e598283-d5e4-489e-a446-3a4fca30fb56","Type":"ContainerStarted","Data":"302546234ab8d18284155e0564f7d86a42a87c3077abc997d581e014a69f1d4e"}
Jul 13 21:10:20.226377 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.226326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:20.563854 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.563815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:10:20.564033 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.563876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:10:20.564033 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.563903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:10:20.566482 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.566453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd368277-a7bf-4c9e-bdcc-90aea0f41dbb-metrics-tls\") pod \"dns-default-s2fq8\" (UID: \"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb\") " pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:10:20.566656 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.566638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"image-registry-5ccd5cb9f6-z8v8f\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") " pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:10:20.566729 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.566675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e278d43d-f638-4047-96ca-a5ef535138d7-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wc5th\" (UID: \"e278d43d-f638-4047-96ca-a5ef535138d7\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:10:20.764957 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.764931    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2l9mp\""
Jul 13 21:10:20.772829 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.772812    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:10:20.782783 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.782764    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t7s7s\""
Jul 13 21:10:20.790643 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.790622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th"
Jul 13 21:10:20.804927 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.804821    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gqhqf\""
Jul 13 21:10:20.813737 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.813714    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:10:20.935149 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.933984    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8jrfk" podStartSLOduration=66.933967583 podStartE2EDuration="1m6.933967583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:19.176571739 +0000 UTC m=+65.834228745" watchObservedRunningTime="2026-07-13 21:10:20.933967583 +0000 UTC m=+67.591624588"
Jul 13 21:10:20.935319 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.935277    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"]
Jul 13 21:10:20.941072 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:10:20.941045    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2281219e_2726_490f_80cb_cb963347c09e.slice/crio-01a6b1574ba5ef53c991cf1bfccab6bb4e2182273645029bab9cf698bddaa4e4 WatchSource:0}: Error finding container 01a6b1574ba5ef53c991cf1bfccab6bb4e2182273645029bab9cf698bddaa4e4: Status 404 returned error can't find the container with id 01a6b1574ba5ef53c991cf1bfccab6bb4e2182273645029bab9cf698bddaa4e4
Jul 13 21:10:20.950190 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.950163    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-wc5th"]
Jul 13 21:10:20.953437 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:10:20.953399    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode278d43d_f638_4047_96ca_a5ef535138d7.slice/crio-6d89f0fc3fb832d96dd71dadfeb95991211a4e0eeb95df9be4161faab26c49f6 WatchSource:0}: Error finding container 6d89f0fc3fb832d96dd71dadfeb95991211a4e0eeb95df9be4161faab26c49f6: Status 404 returned error can't find the container with id 6d89f0fc3fb832d96dd71dadfeb95991211a4e0eeb95df9be4161faab26c49f6
Jul 13 21:10:20.984485 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:20.984459    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s2fq8"]
Jul 13 21:10:20.988996 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:10:20.988895    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd368277_a7bf_4c9e_bdcc_90aea0f41dbb.slice/crio-ebe52c998932524a67bc31c302f7219d1ca49ad97689defaf304aa2a1122e1b8 WatchSource:0}: Error finding container ebe52c998932524a67bc31c302f7219d1ca49ad97689defaf304aa2a1122e1b8: Status 404 returned error can't find the container with id ebe52c998932524a67bc31c302f7219d1ca49ad97689defaf304aa2a1122e1b8
Jul 13 21:10:21.175410 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:21.175316    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s2fq8" event={"ID":"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb","Type":"ContainerStarted","Data":"ebe52c998932524a67bc31c302f7219d1ca49ad97689defaf304aa2a1122e1b8"}
Jul 13 21:10:21.176825 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:21.176794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th" event={"ID":"e278d43d-f638-4047-96ca-a5ef535138d7","Type":"ContainerStarted","Data":"6d89f0fc3fb832d96dd71dadfeb95991211a4e0eeb95df9be4161faab26c49f6"}
Jul 13 21:10:21.178993 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:21.178931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fwj6w" event={"ID":"6e598283-d5e4-489e-a446-3a4fca30fb56","Type":"ContainerStarted","Data":"a988d5ee4951ffaa663f0fa6494362d80fcb3a864c0d0dad5ac50bb4de50ca40"}
Jul 13 21:10:21.180806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:21.180768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f" event={"ID":"2281219e-2726-490f-80cb-cb963347c09e","Type":"ContainerStarted","Data":"f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6"}
Jul 13 21:10:21.180928 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:21.180817    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f" event={"ID":"2281219e-2726-490f-80cb-cb963347c09e","Type":"ContainerStarted","Data":"01a6b1574ba5ef53c991cf1bfccab6bb4e2182273645029bab9cf698bddaa4e4"}
Jul 13 21:10:21.180989 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:21.180946    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:10:21.195202 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:21.195141    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fwj6w" podStartSLOduration=66.206311716 podStartE2EDuration="1m7.195121921s" totalImagesPullingTime="988.810205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:21.194661468 +0000 UTC m=+67.852318477" watchObservedRunningTime="2026-07-13 21:10:21.195121921 +0000 UTC m=+67.852778929"
Jul 13 21:10:21.211321 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:21.211278    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f" podStartSLOduration=66.211264026 podStartE2EDuration="1m6.211264026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:21.210895037 +0000 UTC m=+67.868552044" watchObservedRunningTime="2026-07-13 21:10:21.211264026 +0000 UTC m=+67.868921036"
Jul 13 21:10:23.195104 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:23.194628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th" event={"ID":"e278d43d-f638-4047-96ca-a5ef535138d7","Type":"ContainerStarted","Data":"93e619d315cf41809d70e9688207cdd1986baee43b4012467c7041bd23d8c866"}
Jul 13 21:10:23.882214 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:23.882151    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-wc5th" podStartSLOduration=40.210964851 podStartE2EDuration="41.882130355s" totalImagesPullingTime="1.671165504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:23.211881568 +0000 UTC m=+69.869538572" watchObservedRunningTime="2026-07-13 21:10:23.882130355 +0000 UTC m=+70.539787362"
Jul 13 21:10:23.883096 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:23.883075    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"]
Jul 13 21:10:23.997794 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:23.997763    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:24.095947 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:24.095913    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:24.203200 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:24.203110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:24.329520 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:24.329489    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:24.392242 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:24.392206    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:25.133960 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:25.133927    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:25.207943 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:25.207910    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xl4b6" event={"ID":"166204bf-e576-42fa-80df-02d81b311427","Type":"ContainerStarted","Data":"9dc66a71ffdf357698d4de309c8a1da0cb079b2df50bf76a59544357b781d65d"}
Jul 13 21:10:25.210307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:25.210278    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s2fq8" event={"ID":"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb","Type":"ContainerStarted","Data":"bd8ba815f520b95256d3c01734dbc71f99f1b775fbf3fa7d947d710871090356"}
Jul 13 21:10:25.226437 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:25.226382    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xl4b6" podStartSLOduration=65.199893504 podStartE2EDuration="1m11.226365009s" totalImagesPullingTime="6.026471505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:25.224724389 +0000 UTC m=+71.882381390" watchObservedRunningTime="2026-07-13 21:10:25.226365009 +0000 UTC m=+71.884022016"
Jul 13 21:10:25.508984 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:25.508949    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:25.610710 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:25.610676    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:25.800761 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:25.800694    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:25.858217 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:25.858182    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:26.216624 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:26.216523    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s2fq8" event={"ID":"fd368277-a7bf-4c9e-bdcc-90aea0f41dbb","Type":"ContainerStarted","Data":"eebabe89613154c2a2f19c1b8866387aee6208b6520f1d67daf4b1cb74e25543"}
Jul 13 21:10:26.217040 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:26.216716    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:10:26.237152 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:26.237095    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s2fq8" podStartSLOduration=34.207402078 podStartE2EDuration="38.237077092s" totalImagesPullingTime="4.029675014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:26.236215897 +0000 UTC m=+72.893872905" watchObservedRunningTime="2026-07-13 21:10:26.237077092 +0000 UTC m=+72.894734098"
Jul 13 21:10:28.835056 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:28.835025    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:28.888217 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:28.888186    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:29.011297 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:29.011270    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:32.070476 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:32.070442    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:32.145362 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:32.145330    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:32.530491 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:32.530449    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:32.628745 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:32.628715    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:32.730178 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:32.730136    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:33.175435 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.175401    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:33.248793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.248770    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:33.348978 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.348946    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:33.456947 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.456895    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:33.659931 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.659905    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gbwwc"]
Jul 13 21:10:33.673939 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.673921    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.677543 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.677421    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gbwwc"]
Jul 13 21:10:33.680318 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.680303    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 21:10:33.680771 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.680742    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gz76z\""
Jul 13 21:10:33.681459 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.681435    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 21:10:33.681902 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.681886    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 21:10:33.682333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.682318    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 21:10:33.686324 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.686306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:33.747939 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.747918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9hb4\" (UniqueName: \"kubernetes.io/projected/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-api-access-q9hb4\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.748040 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.747957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.748040 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.747986    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.748127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.748053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f99a209-3a7e-47c8-b6b6-c7c498775028-crio-socket\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.748127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.748121    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f99a209-3a7e-47c8-b6b6-c7c498775028-data-volume\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.751760 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.751739    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:33.839218 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.839187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:33.849092 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.849068    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f99a209-3a7e-47c8-b6b6-c7c498775028-data-volume\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.849189 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.849102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9hb4\" (UniqueName: \"kubernetes.io/projected/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-api-access-q9hb4\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.849189 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.849128    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.849189 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.849148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.849189 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.849163    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f99a209-3a7e-47c8-b6b6-c7c498775028-crio-socket\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.849448 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:33.849238    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 21:10:33.849448 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.849240    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f99a209-3a7e-47c8-b6b6-c7c498775028-crio-socket\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.849448 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:33.849309    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm podName:5f99a209-3a7e-47c8-b6b6-c7c498775028 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:34.349295035 +0000 UTC m=+81.006952018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gbwwc" (UID: "5f99a209-3a7e-47c8-b6b6-c7c498775028") : configmap "kube-rbac-proxy" not found
Jul 13 21:10:33.849448 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:33.849249    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 21:10:33.849448 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:33.849363    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls podName:5f99a209-3a7e-47c8-b6b6-c7c498775028 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:34.349349984 +0000 UTC m=+81.007006973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gbwwc" (UID: "5f99a209-3a7e-47c8-b6b6-c7c498775028") : secret "insights-runtime-extractor-tls" not found
Jul 13 21:10:33.849448 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.849440    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f99a209-3a7e-47c8-b6b6-c7c498775028-data-volume\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:33.857852 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:33.857836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9hb4\" (UniqueName: \"kubernetes.io/projected/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-api-access-q9hb4\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:34.044128 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:34.044061    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:34.095966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:34.095941    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:34.231546 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:34.231521    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:34.281620 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:34.281572    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:34.353494 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:34.353441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:34.353494 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:34.353481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:34.353625 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:34.353547    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 21:10:34.353625 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:34.353569    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 21:10:34.353693 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:34.353628    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm podName:5f99a209-3a7e-47c8-b6b6-c7c498775028 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:35.353615106 +0000 UTC m=+82.011272090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gbwwc" (UID: "5f99a209-3a7e-47c8-b6b6-c7c498775028") : configmap "kube-rbac-proxy" not found
Jul 13 21:10:34.353693 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:34.353646    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls podName:5f99a209-3a7e-47c8-b6b6-c7c498775028 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:35.35363512 +0000 UTC m=+82.011292105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gbwwc" (UID: "5f99a209-3a7e-47c8-b6b6-c7c498775028") : secret "insights-runtime-extractor-tls" not found
Jul 13 21:10:34.643471 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:34.643406    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:34.847532 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:34.847499    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:35.133653 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:35.133606    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:35.213687 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:35.213660    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:35.362156 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:35.362128    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:35.362563 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:35.362177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:35.362563 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:35.362259    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 21:10:35.362563 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:35.362307    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm podName:5f99a209-3a7e-47c8-b6b6-c7c498775028 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:37.362292175 +0000 UTC m=+84.019949159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gbwwc" (UID: "5f99a209-3a7e-47c8-b6b6-c7c498775028") : configmap "kube-rbac-proxy" not found
Jul 13 21:10:35.362563 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:35.362306    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 21:10:35.362563 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:35.362363    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls podName:5f99a209-3a7e-47c8-b6b6-c7c498775028 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:37.362345729 +0000 UTC m=+84.020002732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gbwwc" (UID: "5f99a209-3a7e-47c8-b6b6-c7c498775028") : secret "insights-runtime-extractor-tls" not found
Jul 13 21:10:35.380031 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:35.380002    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:35.442553 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:35.442485    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s2fq8_fd368277-a7bf-4c9e-bdcc-90aea0f41dbb/dns/0.log"
Jul 13 21:10:35.642115 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:35.642091    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s2fq8_fd368277-a7bf-4c9e-bdcc-90aea0f41dbb/kube-rbac-proxy/0.log"
Jul 13 21:10:35.864089 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:35.864053    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:35.917309 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:35.917281    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:36.221525 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:36.221460    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s2fq8"
Jul 13 21:10:36.642160 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:36.642136    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s4zzn_30ac7b1f-d387-4c5a-9b4a-3c99304f8d45/dns-node-resolver/0.log"
Jul 13 21:10:36.842113 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:36.842087    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5ccd5cb9f6-z8v8f_2281219e-2726-490f-80cb-cb963347c09e/registry/0.log"
Jul 13 21:10:37.207743 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:37.207715    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:37.360063 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:37.360027    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:37.375816 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:37.375792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:37.375914 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:37.375846    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:37.375952 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:37.375934    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 21:10:37.376000 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:37.375991    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls podName:5f99a209-3a7e-47c8-b6b6-c7c498775028 nodeName:}" failed. No retries permitted until 2026-07-13 21:10:41.375978485 +0000 UTC m=+88.033635469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gbwwc" (UID: "5f99a209-3a7e-47c8-b6b6-c7c498775028") : secret "insights-runtime-extractor-tls" not found
Jul 13 21:10:37.376310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:37.376293    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f99a209-3a7e-47c8-b6b6-c7c498775028-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:37.442689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:37.442665    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcjdj_f76f5b2b-3323-4299-9eef-aa9bc1763402/node-ca/0.log"
Jul 13 21:10:38.065267 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:38.065234    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:38.443299 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:38.443219    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m8nwq_a1e66718-f6ca-466b-80f6-ee99fc06444f/serve-healthcheck-canary/0.log"
Jul 13 21:10:39.296999 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.296969    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:39.514088 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.514048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:39.569703 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.569640    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-grslf"]
Jul 13 21:10:39.572877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.572862    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.575264 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.575244    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 21:10:39.575375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.575270    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 21:10:39.575375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.575350    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cwc9r\""
Jul 13 21:10:39.575375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.575350    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 21:10:39.576201 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.576181    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 21:10:39.576201 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.576195    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 21:10:39.576366 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.576309    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 21:10:39.605871 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.605840    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:39.669266 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.669244    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:39.691313 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.691289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-wtmp\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.691384 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.691341    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2afea579-811d-46ea-8cd9-b7561d66efcb-root\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.691384 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.691362    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2afea579-811d-46ea-8cd9-b7561d66efcb-metrics-client-ca\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.691460 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.691397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-textfile\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.691460 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.691416    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-accelerators-collector-config\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.691460 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.691433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4vq\" (UniqueName: \"kubernetes.io/projected/2afea579-811d-46ea-8cd9-b7561d66efcb-kube-api-access-7t4vq\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.691460 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.691448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-tls\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.691601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.691465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2afea579-811d-46ea-8cd9-b7561d66efcb-sys\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.691601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.691516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.787693 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.787669    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:39.792605 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2afea579-811d-46ea-8cd9-b7561d66efcb-root\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.792673 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2afea579-811d-46ea-8cd9-b7561d66efcb-metrics-client-ca\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.792673 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-textfile\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.792673 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-accelerators-collector-config\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.792824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4vq\" (UniqueName: \"kubernetes.io/projected/2afea579-811d-46ea-8cd9-b7561d66efcb-kube-api-access-7t4vq\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.792824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2afea579-811d-46ea-8cd9-b7561d66efcb-root\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.792824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792704    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-tls\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.792824 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:39.792777    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 13 21:10:39.792824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2afea579-811d-46ea-8cd9-b7561d66efcb-sys\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.793052 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:39.792833    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-tls podName:2afea579-811d-46ea-8cd9-b7561d66efcb nodeName:}" failed. No retries permitted until 2026-07-13 21:10:40.292817057 +0000 UTC m=+86.950474040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-tls") pod "node-exporter-grslf" (UID: "2afea579-811d-46ea-8cd9-b7561d66efcb") : secret "node-exporter-tls" not found
Jul 13 21:10:39.793052 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.793052 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792863    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2afea579-811d-46ea-8cd9-b7561d66efcb-sys\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.793052 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-wtmp\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.793052 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.792991    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-textfile\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.793207 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.793061    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-wtmp\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.793260 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.793244    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-accelerators-collector-config\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.793302 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.793288    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2afea579-811d-46ea-8cd9-b7561d66efcb-metrics-client-ca\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.795269 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.795249    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.800752 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.800722    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4vq\" (UniqueName: \"kubernetes.io/projected/2afea579-811d-46ea-8cd9-b7561d66efcb-kube-api-access-7t4vq\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:39.888705 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.888645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:39.988838 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:39.988814    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:40.060713 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:40.060687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:40.296915 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:40.296879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-tls\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:40.299227 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:40.299210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2afea579-811d-46ea-8cd9-b7561d66efcb-node-exporter-tls\") pod \"node-exporter-grslf\" (UID: \"2afea579-811d-46ea-8cd9-b7561d66efcb\") " pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:40.482154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:40.482127    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-grslf"
Jul 13 21:10:40.492527 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:10:40.492506    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afea579_811d_46ea_8cd9_b7561d66efcb.slice/crio-150cc0112e8b0508b6212bca99e4035b2acf12c8a4aea403bf9186752e476e50 WatchSource:0}: Error finding container 150cc0112e8b0508b6212bca99e4035b2acf12c8a4aea403bf9186752e476e50: Status 404 returned error can't find the container with id 150cc0112e8b0508b6212bca99e4035b2acf12c8a4aea403bf9186752e476e50
Jul 13 21:10:40.519924 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:40.519900    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:40.603271 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:40.603218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:40.856574 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:40.856504    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:41.269173 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:41.269125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-grslf" event={"ID":"2afea579-811d-46ea-8cd9-b7561d66efcb","Type":"ContainerStarted","Data":"150cc0112e8b0508b6212bca99e4035b2acf12c8a4aea403bf9186752e476e50"}
Jul 13 21:10:41.404148 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:41.404103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:41.406871 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:41.406843    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f99a209-3a7e-47c8-b6b6-c7c498775028-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gbwwc\" (UID: \"5f99a209-3a7e-47c8-b6b6-c7c498775028\") " pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:41.494527 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:41.494500    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gbwwc"
Jul 13 21:10:41.642457 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:41.642429    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gbwwc"]
Jul 13 21:10:41.645647 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:10:41.645617    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f99a209_3a7e_47c8_b6b6_c7c498775028.slice/crio-8267d7d55dc6ee6191aba3f9ef605f571b22f60ac1faf51293e6bc493bf6fb19 WatchSource:0}: Error finding container 8267d7d55dc6ee6191aba3f9ef605f571b22f60ac1faf51293e6bc493bf6fb19: Status 404 returned error can't find the container with id 8267d7d55dc6ee6191aba3f9ef605f571b22f60ac1faf51293e6bc493bf6fb19
Jul 13 21:10:42.277610 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:42.277178    2564 generic.go:356] "Generic (PLEG): container finished" podID="2afea579-811d-46ea-8cd9-b7561d66efcb" containerID="014654814aeca79512ad39e6bf200a288cc805b7897d201fbd7c65a79b310f88" exitCode=0
Jul 13 21:10:42.277610 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:42.277325    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-grslf" event={"ID":"2afea579-811d-46ea-8cd9-b7561d66efcb","Type":"ContainerDied","Data":"014654814aeca79512ad39e6bf200a288cc805b7897d201fbd7c65a79b310f88"}
Jul 13 21:10:42.279987 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:42.279897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gbwwc" event={"ID":"5f99a209-3a7e-47c8-b6b6-c7c498775028","Type":"ContainerStarted","Data":"8267d7d55dc6ee6191aba3f9ef605f571b22f60ac1faf51293e6bc493bf6fb19"}
Jul 13 21:10:42.470854 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:42.470633    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:42.610415 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:42.610335    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:42.686045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:42.686009    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:42.748337 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:42.748305    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:43.286391 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:43.286352    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-grslf" event={"ID":"2afea579-811d-46ea-8cd9-b7561d66efcb","Type":"ContainerStarted","Data":"c1d16a88554b17715b9860f43ed132b6fc074015d69eaef853a39a3f52f75dc2"}
Jul 13 21:10:43.286552 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:43.286418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-grslf" event={"ID":"2afea579-811d-46ea-8cd9-b7561d66efcb","Type":"ContainerStarted","Data":"bff6ca03527878f41d15103695633e6c91ae298af4cae8f202508d9743017eec"}
Jul 13 21:10:43.311993 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:43.311935    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-grslf" podStartSLOduration=3.189657398 podStartE2EDuration="4.311917244s" totalImagesPullingTime="1.046172281s" totalInitContainerRuntime="76.087565ms" isStatefulPod=true podCreationTimestamp="2026-07-13 21:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:43.311610091 +0000 UTC m=+89.969267098" watchObservedRunningTime="2026-07-13 21:10:43.311917244 +0000 UTC m=+89.969574457"
Jul 13 21:10:43.890218 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:43.890188    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:10:44.325064 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:44.324293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:44.384779 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:44.384743    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:44.669201 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:44.669109    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:45.341519 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:45.341481    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:45.442599 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:45.442561    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:45.736442 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:45.736344    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:45.868686 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:45.868659    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:45.925089 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:45.925057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:46.301090 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:46.301058    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gbwwc" event={"ID":"5f99a209-3a7e-47c8-b6b6-c7c498775028","Type":"ContainerStarted","Data":"5838c77706ba7ff1a8bf38fe405bfc01c51fccb63343ad14cb9e92addc3dbea7"}
Jul 13 21:10:46.301257 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:46.301119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gbwwc" event={"ID":"5f99a209-3a7e-47c8-b6b6-c7c498775028","Type":"ContainerStarted","Data":"d41fde4b161ab96cd7bede5cdbceba81824b97c7ebdd85be847b3afea9a9af0e"}
Jul 13 21:10:46.469674 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:46.469634    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:46.521893 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:46.521858    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:47.661258 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:47.661159    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:47.761676 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:47.761640    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:48.907281 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:48.907239    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f" podUID="2281219e-2726-490f-80cb-cb963347c09e" containerName="registry" containerID="cri-o://f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6" gracePeriod=30
Jul 13 21:10:49.065547 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.065518    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:49.152080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.152057    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:10:49.266284 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266262    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-registry-certificates\") pod \"2281219e-2726-490f-80cb-cb963347c09e\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") "
Jul 13 21:10:49.266396 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266292    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-trusted-ca\") pod \"2281219e-2726-490f-80cb-cb963347c09e\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") "
Jul 13 21:10:49.266396 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266321    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") pod \"2281219e-2726-490f-80cb-cb963347c09e\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") "
Jul 13 21:10:49.266396 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266342    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-bound-sa-token\") pod \"2281219e-2726-490f-80cb-cb963347c09e\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") "
Jul 13 21:10:49.266396 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266372    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-installation-pull-secrets\") pod \"2281219e-2726-490f-80cb-cb963347c09e\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") "
Jul 13 21:10:49.266612 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266515    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-image-registry-private-configuration\") pod \"2281219e-2726-490f-80cb-cb963347c09e\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") "
Jul 13 21:10:49.266612 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266562    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2281219e-2726-490f-80cb-cb963347c09e-ca-trust-extracted\") pod \"2281219e-2726-490f-80cb-cb963347c09e\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") "
Jul 13 21:10:49.266713 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266634    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qnn9\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-kube-api-access-9qnn9\") pod \"2281219e-2726-490f-80cb-cb963347c09e\" (UID: \"2281219e-2726-490f-80cb-cb963347c09e\") "
Jul 13 21:10:49.266809 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266760    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2281219e-2726-490f-80cb-cb963347c09e" (UID: "2281219e-2726-490f-80cb-cb963347c09e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:10:49.266809 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2281219e-2726-490f-80cb-cb963347c09e" (UID: "2281219e-2726-490f-80cb-cb963347c09e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:10:49.267055 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266861    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-registry-certificates\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:10:49.267055 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.266880    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2281219e-2726-490f-80cb-cb963347c09e-trusted-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:10:49.269161 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.269116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-kube-api-access-9qnn9" (OuterVolumeSpecName: "kube-api-access-9qnn9") pod "2281219e-2726-490f-80cb-cb963347c09e" (UID: "2281219e-2726-490f-80cb-cb963347c09e"). InnerVolumeSpecName "kube-api-access-9qnn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:10:49.269270 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.269166    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2281219e-2726-490f-80cb-cb963347c09e" (UID: "2281219e-2726-490f-80cb-cb963347c09e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:10:49.269270 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.269256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2281219e-2726-490f-80cb-cb963347c09e" (UID: "2281219e-2726-490f-80cb-cb963347c09e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:10:49.269358 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.269259    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2281219e-2726-490f-80cb-cb963347c09e" (UID: "2281219e-2726-490f-80cb-cb963347c09e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:10:49.269358 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.269318    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2281219e-2726-490f-80cb-cb963347c09e" (UID: "2281219e-2726-490f-80cb-cb963347c09e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:10:49.277498 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.277465    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2281219e-2726-490f-80cb-cb963347c09e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2281219e-2726-490f-80cb-cb963347c09e" (UID: "2281219e-2726-490f-80cb-cb963347c09e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:10:49.313728 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.313701    2564 generic.go:356] "Generic (PLEG): container finished" podID="2281219e-2726-490f-80cb-cb963347c09e" containerID="f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6" exitCode=0
Jul 13 21:10:49.313834 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.313762    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"
Jul 13 21:10:49.313834 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.313788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f" event={"ID":"2281219e-2726-490f-80cb-cb963347c09e","Type":"ContainerDied","Data":"f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6"}
Jul 13 21:10:49.313904 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.313859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f" event={"ID":"2281219e-2726-490f-80cb-cb963347c09e","Type":"ContainerDied","Data":"01a6b1574ba5ef53c991cf1bfccab6bb4e2182273645029bab9cf698bddaa4e4"}
Jul 13 21:10:49.313904 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.313883    2564 scope.go:117] "RemoveContainer" containerID="f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6"
Jul 13 21:10:49.316419 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.316348    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gbwwc" event={"ID":"5f99a209-3a7e-47c8-b6b6-c7c498775028","Type":"ContainerStarted","Data":"2d1d4ba4fd1a90343c5794627a3959d36cc9dd4587d4132c7462d98c4c423acb"}
Jul 13 21:10:49.324605 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.324525    2564 scope.go:117] "RemoveContainer" containerID="f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6"
Jul 13 21:10:49.324840 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:10:49.324812    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6\": container with ID starting with f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6 not found: ID does not exist" containerID="f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6"
Jul 13 21:10:49.324908 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.324851    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6"} err="failed to get container status \"f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6\": rpc error: code = NotFound desc = could not find container \"f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6\": container with ID starting with f5a5ba28f05f6fe50e89b601bee2e6d8e8057862e6e7964391a60903460957f6 not found: ID does not exist"
Jul 13 21:10:49.336890 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.336851    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gbwwc" podStartSLOduration=9.764541222 podStartE2EDuration="16.336840327s" totalImagesPullingTime="6.572299105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:10:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:10:49.336134941 +0000 UTC m=+95.993791948" watchObservedRunningTime="2026-07-13 21:10:49.336840327 +0000 UTC m=+95.994497334"
Jul 13 21:10:49.352070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.352043    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"]
Jul 13 21:10:49.355266 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.355248    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5ccd5cb9f6-z8v8f"]
Jul 13 21:10:49.367605 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.367564    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnn9\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-kube-api-access-9qnn9\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:10:49.367700 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.367609    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-registry-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:10:49.367700 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.367624    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2281219e-2726-490f-80cb-cb963347c09e-bound-sa-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:10:49.367700 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.367636    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-installation-pull-secrets\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:10:49.367700 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.367649    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2281219e-2726-490f-80cb-cb963347c09e-image-registry-private-configuration\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:10:49.367700 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.367661    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2281219e-2726-490f-80cb-cb963347c09e-ca-trust-extracted\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:10:49.820296 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.820269    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2281219e-2726-490f-80cb-cb963347c09e" path="/var/lib/kubelet/pods/2281219e-2726-490f-80cb-cb963347c09e/volumes"
Jul 13 21:10:49.984241 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:49.984207    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:50.084502 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:50.084442    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:50.171912 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:50.171888    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8jrfk"
Jul 13 21:10:53.487817 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:53.487782    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:53.590786 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:53.590753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:53.882214 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:53.882140    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:54.361066 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:54.361035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:54.465395 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:54.465369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:56.140939 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:56.140898    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:56.221937 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:56.221908    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:57.347734 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:57.347705    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:57.446865 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:57.446835    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:57.502686 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:57.502659    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:59.046782 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:59.046747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:10:59.183574 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:10:59.183541    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:02.709942 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:02.709903    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:02.767671 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:02.767645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:02.949404 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:02.949377    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:03.049629 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:03.049601    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:03.113880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:03.113849    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:03.661305 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:03.661273    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:03.713398 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:03.713369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:05.726220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:05.726178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:13.414470 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:13.414437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:13.515537 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:13.515503    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:13.570491 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:13.570465    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:17.657313 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:17.657272    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:17.757597 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:17.757562    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:22.428049 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:22.428018    2564 generic.go:356] "Generic (PLEG): container finished" podID="9dfd688a-ca7d-4474-8d60-5e0760699a39" containerID="1ee2db52500f1e0cebad68001f6775bc220e975af9919835606381c7e1ba233c" exitCode=0
Jul 13 21:11:22.428415 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:22.428101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx" event={"ID":"9dfd688a-ca7d-4474-8d60-5e0760699a39","Type":"ContainerDied","Data":"1ee2db52500f1e0cebad68001f6775bc220e975af9919835606381c7e1ba233c"}
Jul 13 21:11:22.428415 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:22.428402    2564 scope.go:117] "RemoveContainer" containerID="1ee2db52500f1e0cebad68001f6775bc220e975af9919835606381c7e1ba233c"
Jul 13 21:11:23.432772 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:23.432741    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-bxdrx" event={"ID":"9dfd688a-ca7d-4474-8d60-5e0760699a39","Type":"ContainerStarted","Data":"e9ceac3682962985aac6d3a165c01b10e67923814e79d084f2851e614dfb08cc"}
Jul 13 21:11:24.996062 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:24.996022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:25.095820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:25.095788    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:25.197386 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:25.197352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:31.340854 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:31.340808    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" podUID="858ce0cc-f7ad-4532-a2e1-dea2ccced699" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 13 21:11:41.342167 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:41.342128    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" podUID="858ce0cc-f7ad-4532-a2e1-dea2ccced699" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 13 21:11:51.341536 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:51.341493    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" podUID="858ce0cc-f7ad-4532-a2e1-dea2ccced699" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 13 21:11:51.341973 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:51.341604    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs"
Jul 13 21:11:51.342120 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:51.342100    2564 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c01f869ce440d12feefdc4023c545da973ad80b444bf2af35f185e86a6bb5d1e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 13 21:11:51.342160 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:51.342143    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" podUID="858ce0cc-f7ad-4532-a2e1-dea2ccced699" containerName="service-proxy" containerID="cri-o://c01f869ce440d12feefdc4023c545da973ad80b444bf2af35f185e86a6bb5d1e" gracePeriod=30
Jul 13 21:11:51.526197 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:51.526155    2564 generic.go:356] "Generic (PLEG): container finished" podID="858ce0cc-f7ad-4532-a2e1-dea2ccced699" containerID="c01f869ce440d12feefdc4023c545da973ad80b444bf2af35f185e86a6bb5d1e" exitCode=2
Jul 13 21:11:51.526361 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:51.526214    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" event={"ID":"858ce0cc-f7ad-4532-a2e1-dea2ccced699","Type":"ContainerDied","Data":"c01f869ce440d12feefdc4023c545da973ad80b444bf2af35f185e86a6bb5d1e"}
Jul 13 21:11:51.526361 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:51.526272    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6975fb5dc7-bpdgs" event={"ID":"858ce0cc-f7ad-4532-a2e1-dea2ccced699","Type":"ContainerStarted","Data":"580ad197d3e82ef362c3e256f51f1c1a1c229a33cb54493b3f3bda60d76ea5ab"}
Jul 13 21:11:55.623505 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:55.623474    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:11:55.683497 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:11:55.683465    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:03.917336 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:03.917299    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:03.971192 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:03.971160    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:05.225064 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:05.224980    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:05.292744 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:05.292716    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:05.367708 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:05.367678    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:05.426325 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:05.426284    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:05.962019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:05.961984    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:07.489576 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:07.489542    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:07.589911 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:07.589872    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:15.642652 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:15.642618    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:15.703847 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:15.703808    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:15.763103 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:15.763071    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:17.662981 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:17.662947    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:12:17.762408 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:12:17.762377    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:02.756238 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:02.756203    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:02.944766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:02.944729    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:29.455421 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:29.455385    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:30.206820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:30.206789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:33.039058 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:33.039021    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:33.139684 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:33.139648    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:35.161410 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:35.161326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:35.285643 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:35.285604    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:37.520102 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:37.520058    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:37.729003 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:37.728974    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:39.840764 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:39.840735    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:39.940165 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:39.940136    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:44.179715 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:44.179677    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:44.233453 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:44.233426    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:45.130056 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:45.130025    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:46.252569 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:46.252534    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:46.349293 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:46.349261    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:48.264313 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.264271    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"]
Jul 13 21:13:48.264736 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.264557    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2281219e-2726-490f-80cb-cb963347c09e" containerName="registry"
Jul 13 21:13:48.264736 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.264569    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2281219e-2726-490f-80cb-cb963347c09e" containerName="registry"
Jul 13 21:13:48.264736 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.264641    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2281219e-2726-490f-80cb-cb963347c09e" containerName="registry"
Jul 13 21:13:48.267499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.267478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"
Jul 13 21:13:48.269994 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.269972    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rmqd6\""
Jul 13 21:13:48.270099 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.270036    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 13 21:13:48.270099 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.270037    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 13 21:13:48.276231 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.276200    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"]
Jul 13 21:13:48.432620 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.432571    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vsdh2\" (UID: \"d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"
Jul 13 21:13:48.432736 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.432647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftqd6\" (UniqueName: \"kubernetes.io/projected/d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd-kube-api-access-ftqd6\") pod \"openshift-lws-operator-fd8ccff4c-vsdh2\" (UID: \"d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"
Jul 13 21:13:48.533550 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.533484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftqd6\" (UniqueName: \"kubernetes.io/projected/d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd-kube-api-access-ftqd6\") pod \"openshift-lws-operator-fd8ccff4c-vsdh2\" (UID: \"d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"
Jul 13 21:13:48.533550 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.533547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vsdh2\" (UID: \"d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"
Jul 13 21:13:48.533898 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.533875    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vsdh2\" (UID: \"d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"
Jul 13 21:13:48.541933 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.541909    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftqd6\" (UniqueName: \"kubernetes.io/projected/d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd-kube-api-access-ftqd6\") pod \"openshift-lws-operator-fd8ccff4c-vsdh2\" (UID: \"d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"
Jul 13 21:13:48.577313 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.577290    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"
Jul 13 21:13:48.705805 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.705773    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2"]
Jul 13 21:13:48.708708 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:13:48.708683    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5850608_d7f8_4a3a_8d5a_d17d69c4ddfd.slice/crio-01cae3a318ad29f65991e88430d179f4622c7aecddfe173927cc4a0aebdb4270 WatchSource:0}: Error finding container 01cae3a318ad29f65991e88430d179f4622c7aecddfe173927cc4a0aebdb4270: Status 404 returned error can't find the container with id 01cae3a318ad29f65991e88430d179f4622c7aecddfe173927cc4a0aebdb4270
Jul 13 21:13:48.923784 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:48.923717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2" event={"ID":"d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd","Type":"ContainerStarted","Data":"01cae3a318ad29f65991e88430d179f4622c7aecddfe173927cc4a0aebdb4270"}
Jul 13 21:13:51.937072 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:51.937031    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2" event={"ID":"d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd","Type":"ContainerStarted","Data":"a875f3c71e2c0d531f516ab1cf9f1382d93ed785c88777ac6ef2bb3fd13c7dca"}
Jul 13 21:13:51.952860 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:51.952802    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsdh2" podStartSLOduration=1.341690635 podStartE2EDuration="3.952788412s" totalImagesPullingTime="2.611097777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:13:51.951331241 +0000 UTC m=+278.608988247" watchObservedRunningTime="2026-07-13 21:13:51.952788412 +0000 UTC m=+278.610445418"
Jul 13 21:13:52.609617 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:52.609561    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:52.710088 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:52.710054    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:13:54.316743 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:13:54.316710    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:07.051272 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:07.051238    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:07.151202 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:07.151169    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:07.664967 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:07.664935    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:07.857827 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:07.857794    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:07.960526 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:07.960454    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:08.002169 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.002146    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"]
Jul 13 21:14:08.005308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.005287    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.014550 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.014531    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 13 21:14:08.014838 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.014819    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8vjt2\""
Jul 13 21:14:08.014934 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.014919    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 13 21:14:08.016007 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.015994    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 13 21:14:08.016204 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.016189    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 13 21:14:08.057358 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.057331    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"]
Jul 13 21:14:08.060431 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.060412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bd4b197e-9847-4de7-b873-75a5bb23822a-manifests\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.060519 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.060446    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd4b197e-9847-4de7-b873-75a5bb23822a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.060519 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.060487    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd4b197e-9847-4de7-b873-75a5bb23822a-tmp\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.060641 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.060531    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bd4b197e-9847-4de7-b873-75a5bb23822a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.060641 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.060552    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkrr\" (UniqueName: \"kubernetes.io/projected/bd4b197e-9847-4de7-b873-75a5bb23822a-kube-api-access-vrkrr\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.161038 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.161007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd4b197e-9847-4de7-b873-75a5bb23822a-tmp\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.161139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.161047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bd4b197e-9847-4de7-b873-75a5bb23822a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.161139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.161072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkrr\" (UniqueName: \"kubernetes.io/projected/bd4b197e-9847-4de7-b873-75a5bb23822a-kube-api-access-vrkrr\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.161139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.161106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bd4b197e-9847-4de7-b873-75a5bb23822a-manifests\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.161294 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.161153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd4b197e-9847-4de7-b873-75a5bb23822a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.161486 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.161463    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bd4b197e-9847-4de7-b873-75a5bb23822a-manifests\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.163472 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.163449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd4b197e-9847-4de7-b873-75a5bb23822a-tmp\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.163728 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.163706    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd4b197e-9847-4de7-b873-75a5bb23822a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.163779 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.163750    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bd4b197e-9847-4de7-b873-75a5bb23822a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.171668 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.171644    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkrr\" (UniqueName: \"kubernetes.io/projected/bd4b197e-9847-4de7-b873-75a5bb23822a-kube-api-access-vrkrr\") pod \"opendatahub-operator-controller-manager-84fb46456c-qwbmx\" (UID: \"bd4b197e-9847-4de7-b873-75a5bb23822a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.191992 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.191966    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:08.292600 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.292561    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:08.315177 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.315163    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:08.392696 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.392663    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:08.455882 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.455853    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"]
Jul 13 21:14:08.459713 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:14:08.459688    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4b197e_9847_4de7_b873_75a5bb23822a.slice/crio-dcef8327b36c359ded56cf6ad6b5f140ca8825b42a6f091f51053ab0783df4a6 WatchSource:0}: Error finding container dcef8327b36c359ded56cf6ad6b5f140ca8825b42a6f091f51053ab0783df4a6: Status 404 returned error can't find the container with id dcef8327b36c359ded56cf6ad6b5f140ca8825b42a6f091f51053ab0783df4a6
Jul 13 21:14:08.465416 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.465391    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:08.528267 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.528242    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:08.636446 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.636392    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:08.725991 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.725967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:08.995693 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:08.995667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx" event={"ID":"bd4b197e-9847-4de7-b873-75a5bb23822a","Type":"ContainerStarted","Data":"dcef8327b36c359ded56cf6ad6b5f140ca8825b42a6f091f51053ab0783df4a6"}
Jul 13 21:14:12.009686 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:12.009633    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx" event={"ID":"bd4b197e-9847-4de7-b873-75a5bb23822a","Type":"ContainerStarted","Data":"210fa996cbf0a761d283a0cd150d00ef260861e2ab507f7581f439247a2572d5"}
Jul 13 21:14:12.586607 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:12.586520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:12.688260 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:12.688214    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:13.014883 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:13.014852    2564 generic.go:356] "Generic (PLEG): container finished" podID="bd4b197e-9847-4de7-b873-75a5bb23822a" containerID="210fa996cbf0a761d283a0cd150d00ef260861e2ab507f7581f439247a2572d5" exitCode=0
Jul 13 21:14:13.015244 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:13.014930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx" event={"ID":"bd4b197e-9847-4de7-b873-75a5bb23822a","Type":"ContainerDied","Data":"210fa996cbf0a761d283a0cd150d00ef260861e2ab507f7581f439247a2572d5"}
Jul 13 21:14:13.768061 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:13.768024    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:14:13.768632 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:13.768563    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:14:13.783331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:13.783309    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 21:14:14.020316 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:14.020235    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx" event={"ID":"bd4b197e-9847-4de7-b873-75a5bb23822a","Type":"ContainerStarted","Data":"f02ce7a3afd794060e78b998d0caca373f7c1bcc15bd3d8df4077d8225540010"}
Jul 13 21:14:14.022707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:14.020344    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:14.041292 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:14.041249    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx" podStartSLOduration=2.405945134 podStartE2EDuration="7.041233287s" totalImagesPullingTime="3.622320022s" totalInitContainerRuntime="1.012968131s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:14:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:14:14.03947484 +0000 UTC m=+300.697131847" watchObservedRunningTime="2026-07-13 21:14:14.041233287 +0000 UTC m=+300.698890338"
Jul 13 21:14:15.634383 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:15.634343    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:15.692215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:15.692193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:19.206654 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:19.206611    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:20.569603 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:20.569561    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:25.025427 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:25.025403    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84fb46456c-qwbmx"
Jul 13 21:14:27.182418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.182384    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:27.222106 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.222076    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"]
Jul 13 21:14:27.233807 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.233782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.237222 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.237191    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sn56t\""
Jul 13 21:14:27.237334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.237240    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 13 21:14:27.237334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.237318    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 13 21:14:27.237479 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.237456    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 13 21:14:27.247921 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.247902    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"]
Jul 13 21:14:27.279317 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.279280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:27.295458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.295438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd0e308-c964-4d14-8c4a-f305ec1292e9-cert\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.295544 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.295477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfgqv\" (UniqueName: \"kubernetes.io/projected/5fd0e308-c964-4d14-8c4a-f305ec1292e9-kube-api-access-dfgqv\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.295544 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.295497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5fd0e308-c964-4d14-8c4a-f305ec1292e9-manager-config\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.295637 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.295577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5fd0e308-c964-4d14-8c4a-f305ec1292e9-metrics-cert\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.396814 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.396779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd0e308-c964-4d14-8c4a-f305ec1292e9-cert\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.396970 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.396837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgqv\" (UniqueName: \"kubernetes.io/projected/5fd0e308-c964-4d14-8c4a-f305ec1292e9-kube-api-access-dfgqv\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.396970 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.396869    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5fd0e308-c964-4d14-8c4a-f305ec1292e9-manager-config\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.396970 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.396897    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5fd0e308-c964-4d14-8c4a-f305ec1292e9-metrics-cert\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.397555 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.397530    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5fd0e308-c964-4d14-8c4a-f305ec1292e9-manager-config\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.399526 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.399504    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5fd0e308-c964-4d14-8c4a-f305ec1292e9-metrics-cert\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.399656 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.399570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd0e308-c964-4d14-8c4a-f305ec1292e9-cert\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.416245 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.416225    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgqv\" (UniqueName: \"kubernetes.io/projected/5fd0e308-c964-4d14-8c4a-f305ec1292e9-kube-api-access-dfgqv\") pod \"lws-controller-manager-588f9fd4c6-f957r\" (UID: \"5fd0e308-c964-4d14-8c4a-f305ec1292e9\") " pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.542010 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.541975    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:27.553460 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.553300    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:27.716547 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.716515    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"]
Jul 13 21:14:27.720670 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:14:27.720616    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd0e308_c964_4d14_8c4a_f305ec1292e9.slice/crio-44ab81fbe82e6f180c4d549d446996baa527719b277b9ccc35a5411ff94b95dd WatchSource:0}: Error finding container 44ab81fbe82e6f180c4d549d446996baa527719b277b9ccc35a5411ff94b95dd: Status 404 returned error can't find the container with id 44ab81fbe82e6f180c4d549d446996baa527719b277b9ccc35a5411ff94b95dd
Jul 13 21:14:27.722715 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.722695    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:14:27.991357 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:27.991332    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:28.064643 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:28.064605    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:28.072456 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:28.072419    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r" event={"ID":"5fd0e308-c964-4d14-8c4a-f305ec1292e9","Type":"ContainerStarted","Data":"44ab81fbe82e6f180c4d549d446996baa527719b277b9ccc35a5411ff94b95dd"}
Jul 13 21:14:28.173212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:28.173174    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:28.265604 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:28.265495    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:28.857389 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:28.857360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:29.117027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:29.116943    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:29.187370 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:29.187300    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:29.380937 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:29.380856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:30.060995 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:30.060938    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:30.082478 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:30.082445    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r" event={"ID":"5fd0e308-c964-4d14-8c4a-f305ec1292e9","Type":"ContainerStarted","Data":"6d005f387ac47e11b578e5ad967e267d77722a87375beeb9c4afdd4769f0f393"}
Jul 13 21:14:30.082647 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:30.082533    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:30.099320 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:30.099281    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r" podStartSLOduration=1.578286447 podStartE2EDuration="3.099268668s" totalImagesPullingTime="1.520982221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:14:30.098752255 +0000 UTC m=+316.756409260" watchObservedRunningTime="2026-07-13 21:14:30.099268668 +0000 UTC m=+316.756925673"
Jul 13 21:14:34.984305 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:34.984270    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"]
Jul 13 21:14:34.987157 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:34.987140    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:34.990046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:34.990027    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 13 21:14:34.990540 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:34.990518    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 13 21:14:34.990672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:34.990550    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 13 21:14:34.990672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:34.990560    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dvlnl\""
Jul 13 21:14:34.990672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:34.990598    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 13 21:14:34.990672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:34.990563    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 13 21:14:34.991360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:34.991347    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 13 21:14:35.008162 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.008143    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"]
Jul 13 21:14:35.051178 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.051156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dba4abee-c858-416f-90d1-93430f81c664-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.051268 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.051191    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6fh\" (UniqueName: \"kubernetes.io/projected/dba4abee-c858-416f-90d1-93430f81c664-kube-api-access-gj6fh\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.051268 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.051209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dba4abee-c858-416f-90d1-93430f81c664-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.051268 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.051252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba4abee-c858-416f-90d1-93430f81c664-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.051373 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.051316    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba4abee-c858-416f-90d1-93430f81c664-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.051373 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.051333    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dba4abee-c858-416f-90d1-93430f81c664-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.051373 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.051369    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dba4abee-c858-416f-90d1-93430f81c664-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.152528 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.152499    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dba4abee-c858-416f-90d1-93430f81c664-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.152689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.152540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6fh\" (UniqueName: \"kubernetes.io/projected/dba4abee-c858-416f-90d1-93430f81c664-kube-api-access-gj6fh\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.152689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.152566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dba4abee-c858-416f-90d1-93430f81c664-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.152689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.152613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba4abee-c858-416f-90d1-93430f81c664-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.152689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.152635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba4abee-c858-416f-90d1-93430f81c664-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.152689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.152651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dba4abee-c858-416f-90d1-93430f81c664-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.152907 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.152693    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dba4abee-c858-416f-90d1-93430f81c664-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.153441 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.153364    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dba4abee-c858-416f-90d1-93430f81c664-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.155164 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.155139    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dba4abee-c858-416f-90d1-93430f81c664-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.155164 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.155154    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dba4abee-c858-416f-90d1-93430f81c664-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.155288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.155180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba4abee-c858-416f-90d1-93430f81c664-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.155288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.155206    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dba4abee-c858-416f-90d1-93430f81c664-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.161080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.161054    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba4abee-c858-416f-90d1-93430f81c664-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.161244 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.161230    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6fh\" (UniqueName: \"kubernetes.io/projected/dba4abee-c858-416f-90d1-93430f81c664-kube-api-access-gj6fh\") pod \"istiod-openshift-gateway-94bb8fbfd-mz4h4\" (UID: \"dba4abee-c858-416f-90d1-93430f81c664\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.295578 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.295552    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:35.427181 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:35.427121    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"]
Jul 13 21:14:35.430102 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:14:35.430052    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba4abee_c858_416f_90d1_93430f81c664.slice/crio-3f8b1de794cf019b4c9f4d7d136780d291d8b90e18ba86ddc6d4c456d9b568c9 WatchSource:0}: Error finding container 3f8b1de794cf019b4c9f4d7d136780d291d8b90e18ba86ddc6d4c456d9b568c9: Status 404 returned error can't find the container with id 3f8b1de794cf019b4c9f4d7d136780d291d8b90e18ba86ddc6d4c456d9b568c9
Jul 13 21:14:36.108774 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:36.108743    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4" event={"ID":"dba4abee-c858-416f-90d1-93430f81c664","Type":"ContainerStarted","Data":"3f8b1de794cf019b4c9f4d7d136780d291d8b90e18ba86ddc6d4c456d9b568c9"}
Jul 13 21:14:37.781634 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:37.781565    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:37.858992 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:37.858948    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:37.986475 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:37.986430    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:14:37.986632 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:37.986495    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:14:38.120992 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:38.120955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4" event={"ID":"dba4abee-c858-416f-90d1-93430f81c664","Type":"ContainerStarted","Data":"c5f969bf52880441bff60ebdc41cecbdc54918488896a731827e784fd7a60d07"}
Jul 13 21:14:38.121148 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:38.121083    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:14:38.145870 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:38.145785    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4" podStartSLOduration=1.593301344 podStartE2EDuration="4.145766075s" totalImagesPullingTime="2.552464731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:14:38.143194796 +0000 UTC m=+324.800851803" watchObservedRunningTime="2026-07-13 21:14:38.145766075 +0000 UTC m=+324.803423081"
Jul 13 21:14:38.412933 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:38.412804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:38.518285 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:38.518252    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:38.678617 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:38.678515    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:38.768907 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:38.768869    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:39.127111 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:39.127081    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-mz4h4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 21:14:39.127475 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:39.127134    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4" podUID="dba4abee-c858-416f-90d1-93430f81c664" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:14:39.236785 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:39.236757    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:39.324546 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:39.324513    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:39.518959 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:39.518929    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:39.683016 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:39.682956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:41.089705 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:41.089673    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-588f9fd4c6-f957r"
Jul 13 21:14:41.240808 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:41.240771    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:41.312155 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:41.312125    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:14:42.126682 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:14:42.126657    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mz4h4"
Jul 13 21:15:24.337233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:24.337150    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:25.689559 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:25.689525    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:25.784790 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:25.784759    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:28.656022 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:28.655990    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:28.760319 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:28.760280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:28.824010 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:28.823985    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:29.555763 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:29.555725    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:29.757868 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:29.757828    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:33.357938 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:33.357889    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:33.420295 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:33.420264    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:55.658928 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:55.658880    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:57.660018 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:57.659986    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:57.760121 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:57.760094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:58.106250 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:58.106207    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:15:59.181972 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:15:59.181938    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:02.851145 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:02.851104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:02.958304 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:02.958268    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:03.053127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:03.053099    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:03.150149 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:03.150077    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:03.336203 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:03.336172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:03.611466 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:03.611433    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:03.703700 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:03.703677    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:06.720414 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:06.720353    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:09.318842 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:09.318811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:13.934146 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:13.934094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:14.035853 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:14.035819    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:14.135495 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:14.135467    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:14.233769 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:14.233707    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:15.736839 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:15.736804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:15.837294 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:15.837263    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:16.341289 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:16.341258    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:16.445037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:16.444995    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:22.837688 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:22.837649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:23.313539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:23.313504    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:27.759895 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:27.759856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:28.572568 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:28.572531    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:29.831165 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:29.831134    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:33.179228 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:33.179192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:33.287519 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:33.287492    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:35.933672 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:35.933564    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:36.033882 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:36.033846    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:36.326241 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:36.326209    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:36.504603 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:36.504555    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:39.907694 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:39.907658    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:39.973416 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:39.973384    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:44.055568 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:44.055528    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:44.278481 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:44.278453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:47.507734 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:47.507703    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:47.608804 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:47.608774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:55.643991 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:55.643960    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:55.724187 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:55.724155    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:56.221767 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:56.221728    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:56.292282 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:56.292242    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:59.258230 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:59.258187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:16:59.359010 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:16:59.358977    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:03.814519 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:03.814462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:03.915495 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:03.915459    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:06.391387 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:06.391357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:06.494019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:06.493985    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:20.813086 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:20.813048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:21.488161 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:21.488125    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:24.542510 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:24.542473    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:24.654221 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:24.654184    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:24.710345 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:24.710316    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:25.375314 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:25.375271    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:25.476188 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:25.476153    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:44.322688 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:44.322650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:44.421969 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:44.421943    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:44.766220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:44.766186    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:57.072979 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:57.072923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:17:57.300884 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:17:57.300853    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:01.152218 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:01.152175    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:01.252127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:01.252097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:03.279569 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.279538    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:03.291811 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.288723    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d97c4c679-bh68d"]
Jul 13 21:18:03.293283 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.293254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:03.295763 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.295735    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 13 21:18:03.295932 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.295912    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 13 21:18:03.296045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.296027    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2k6tc\""
Jul 13 21:18:03.301541 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.301522    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d97c4c679-bh68d"]
Jul 13 21:18:03.376766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.376739    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:03.392698 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.392673    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-webhook-cert\") pod \"maas-controller-d97c4c679-bh68d\" (UID: \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\") " pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:03.392782 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.392720    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjmws\" (UniqueName: \"kubernetes.io/projected/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-kube-api-access-hjmws\") pod \"maas-controller-d97c4c679-bh68d\" (UID: \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\") " pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:03.444860 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.444839    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:03.448355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.448337    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66ffdf856b-d9ddr"]
Jul 13 21:18:03.451512 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.451499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:03.459283 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.459262    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66ffdf856b-d9ddr"]
Jul 13 21:18:03.493155 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.493131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-webhook-cert\") pod \"maas-controller-d97c4c679-bh68d\" (UID: \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\") " pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:03.493245 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.493170    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjmws\" (UniqueName: \"kubernetes.io/projected/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-kube-api-access-hjmws\") pod \"maas-controller-d97c4c679-bh68d\" (UID: \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\") " pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:03.493336 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:18:03.493281    2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 13 21:18:03.493377 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:18:03.493353    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-webhook-cert podName:ead8d2af-531c-4918-b18a-9ed3d1bbf7b4 nodeName:}" failed. No retries permitted until 2026-07-13 21:18:03.993332058 +0000 UTC m=+530.650989065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-webhook-cert") pod "maas-controller-d97c4c679-bh68d" (UID: "ead8d2af-531c-4918-b18a-9ed3d1bbf7b4") : secret "maas-controller-webhook-cert" not found
Jul 13 21:18:03.501472 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.501453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjmws\" (UniqueName: \"kubernetes.io/projected/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-kube-api-access-hjmws\") pod \"maas-controller-d97c4c679-bh68d\" (UID: \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\") " pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:03.594097 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.594039    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct6sx\" (UniqueName: \"kubernetes.io/projected/748755b7-b510-4d97-9c8b-c019600831a2-kube-api-access-ct6sx\") pod \"maas-controller-66ffdf856b-d9ddr\" (UID: \"748755b7-b510-4d97-9c8b-c019600831a2\") " pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:03.594190 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.594126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/748755b7-b510-4d97-9c8b-c019600831a2-webhook-cert\") pod \"maas-controller-66ffdf856b-d9ddr\" (UID: \"748755b7-b510-4d97-9c8b-c019600831a2\") " pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:03.695370 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.695347    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/748755b7-b510-4d97-9c8b-c019600831a2-webhook-cert\") pod \"maas-controller-66ffdf856b-d9ddr\" (UID: \"748755b7-b510-4d97-9c8b-c019600831a2\") " pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:03.695453 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.695391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct6sx\" (UniqueName: \"kubernetes.io/projected/748755b7-b510-4d97-9c8b-c019600831a2-kube-api-access-ct6sx\") pod \"maas-controller-66ffdf856b-d9ddr\" (UID: \"748755b7-b510-4d97-9c8b-c019600831a2\") " pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:03.697860 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.697835    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/748755b7-b510-4d97-9c8b-c019600831a2-webhook-cert\") pod \"maas-controller-66ffdf856b-d9ddr\" (UID: \"748755b7-b510-4d97-9c8b-c019600831a2\") " pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:03.703073 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.703051    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct6sx\" (UniqueName: \"kubernetes.io/projected/748755b7-b510-4d97-9c8b-c019600831a2-kube-api-access-ct6sx\") pod \"maas-controller-66ffdf856b-d9ddr\" (UID: \"748755b7-b510-4d97-9c8b-c019600831a2\") " pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:03.727019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.727000    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:03.762560 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.762543    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:03.888375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.888354    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66ffdf856b-d9ddr"]
Jul 13 21:18:03.890865 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:18:03.890837    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748755b7_b510_4d97_9c8b_c019600831a2.slice/crio-833e45fec181c273d4d064bce1a2a09e238a07dfcb9e07268ef05a688c7b97d2 WatchSource:0}: Error finding container 833e45fec181c273d4d064bce1a2a09e238a07dfcb9e07268ef05a688c7b97d2: Status 404 returned error can't find the container with id 833e45fec181c273d4d064bce1a2a09e238a07dfcb9e07268ef05a688c7b97d2
Jul 13 21:18:03.925468 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.925435    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66ffdf856b-d9ddr" event={"ID":"748755b7-b510-4d97-9c8b-c019600831a2","Type":"ContainerStarted","Data":"833e45fec181c273d4d064bce1a2a09e238a07dfcb9e07268ef05a688c7b97d2"}
Jul 13 21:18:03.998156 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:03.998130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-webhook-cert\") pod \"maas-controller-d97c4c679-bh68d\" (UID: \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\") " pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:04.000563 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:04.000547    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-webhook-cert\") pod \"maas-controller-d97c4c679-bh68d\" (UID: \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\") " pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:04.204322 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:04.204260    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:04.330356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:04.330269    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d97c4c679-bh68d"]
Jul 13 21:18:04.332668 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:18:04.332647    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead8d2af_531c_4918_b18a_9ed3d1bbf7b4.slice/crio-474cbede29bb315356d2b95b81cccfc322b127aa08dfaab136afddf60a004bbf WatchSource:0}: Error finding container 474cbede29bb315356d2b95b81cccfc322b127aa08dfaab136afddf60a004bbf: Status 404 returned error can't find the container with id 474cbede29bb315356d2b95b81cccfc322b127aa08dfaab136afddf60a004bbf
Jul 13 21:18:04.934193 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:04.933997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d97c4c679-bh68d" event={"ID":"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4","Type":"ContainerStarted","Data":"474cbede29bb315356d2b95b81cccfc322b127aa08dfaab136afddf60a004bbf"}
Jul 13 21:18:06.949836 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:06.949793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d97c4c679-bh68d" event={"ID":"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4","Type":"ContainerStarted","Data":"4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43"}
Jul 13 21:18:06.950284 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:06.950045    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:06.951739 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:06.951713    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66ffdf856b-d9ddr" event={"ID":"748755b7-b510-4d97-9c8b-c019600831a2","Type":"ContainerStarted","Data":"674c296a7ef45040246facce00eacd210772602bc9f906dea207f76ddcd68107"}
Jul 13 21:18:06.951866 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:06.951840    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:06.966691 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:06.966639    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d97c4c679-bh68d" podStartSLOduration=1.437826 podStartE2EDuration="3.966620685s" totalImagesPullingTime="2.528794685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:18:06.965112234 +0000 UTC m=+533.622769242" watchObservedRunningTime="2026-07-13 21:18:06.966620685 +0000 UTC m=+533.624277693"
Jul 13 21:18:06.980915 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:06.980865    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66ffdf856b-d9ddr" podStartSLOduration=0.99998847 podStartE2EDuration="3.980851589s" totalImagesPullingTime="2.980863119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:18:06.980507961 +0000 UTC m=+533.638164968" watchObservedRunningTime="2026-07-13 21:18:06.980851589 +0000 UTC m=+533.638508595"
Jul 13 21:18:07.084835 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:07.084798    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:07.182535 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:07.182454    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:10.117927 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:10.117898    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:13.609706 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:13.609668    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:17.958201 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:17.958173    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:17.958671 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:17.958243    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:18:18.074452 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:18.074420    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d97c4c679-bh68d"]
Jul 13 21:18:18.074683 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:18.074663    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d97c4c679-bh68d" podUID="ead8d2af-531c-4918-b18a-9ed3d1bbf7b4" containerName="manager" containerID="cri-o://4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43" gracePeriod=30
Jul 13 21:18:18.316432 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:18.316408    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:18.400121 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:18.400092    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-webhook-cert\") pod \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\" (UID: \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\") "
Jul 13 21:18:18.400236 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:18.400159    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjmws\" (UniqueName: \"kubernetes.io/projected/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-kube-api-access-hjmws\") pod \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\" (UID: \"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4\") "
Jul 13 21:18:18.402264 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:18.402238    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-kube-api-access-hjmws" (OuterVolumeSpecName: "kube-api-access-hjmws") pod "ead8d2af-531c-4918-b18a-9ed3d1bbf7b4" (UID: "ead8d2af-531c-4918-b18a-9ed3d1bbf7b4"). InnerVolumeSpecName "kube-api-access-hjmws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:18:18.402362 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:18.402298    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ead8d2af-531c-4918-b18a-9ed3d1bbf7b4" (UID: "ead8d2af-531c-4918-b18a-9ed3d1bbf7b4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:18:18.501491 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:18.501427    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjmws\" (UniqueName: \"kubernetes.io/projected/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-kube-api-access-hjmws\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:18:18.501491 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:18.501457    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4-webhook-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:18:19.006189 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.006151    2564 generic.go:356] "Generic (PLEG): container finished" podID="ead8d2af-531c-4918-b18a-9ed3d1bbf7b4" containerID="4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43" exitCode=0
Jul 13 21:18:19.006608 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.006203    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d97c4c679-bh68d"
Jul 13 21:18:19.006608 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.006235    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d97c4c679-bh68d" event={"ID":"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4","Type":"ContainerDied","Data":"4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43"}
Jul 13 21:18:19.006608 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.006311    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d97c4c679-bh68d" event={"ID":"ead8d2af-531c-4918-b18a-9ed3d1bbf7b4","Type":"ContainerDied","Data":"474cbede29bb315356d2b95b81cccfc322b127aa08dfaab136afddf60a004bbf"}
Jul 13 21:18:19.006608 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.006334    2564 scope.go:117] "RemoveContainer" containerID="4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43"
Jul 13 21:18:19.014756 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.014741    2564 scope.go:117] "RemoveContainer" containerID="4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43"
Jul 13 21:18:19.014941 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:18:19.014925    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43\": container with ID starting with 4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43 not found: ID does not exist" containerID="4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43"
Jul 13 21:18:19.014981 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.014947    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43"} err="failed to get container status \"4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43\": rpc error: code = NotFound desc = could not find container \"4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43\": container with ID starting with 4e14fad5a1da93843b2c12ce7d2254aafb56b6fcb872126b18bce5268b7b4c43 not found: ID does not exist"
Jul 13 21:18:19.026739 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.026701    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d97c4c679-bh68d"]
Jul 13 21:18:19.028645 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.028627    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d97c4c679-bh68d"]
Jul 13 21:18:19.820505 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:19.820472    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead8d2af-531c-4918-b18a-9ed3d1bbf7b4" path="/var/lib/kubelet/pods/ead8d2af-531c-4918-b18a-9ed3d1bbf7b4/volumes"
Jul 13 21:18:39.548228 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:39.548195    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:39.650161 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:39.650120    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:40.104549 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.104521    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"]
Jul 13 21:18:40.104914 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.104889    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:40.105013 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.104987    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead8d2af-531c-4918-b18a-9ed3d1bbf7b4" containerName="manager"
Jul 13 21:18:40.105065 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.105017    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead8d2af-531c-4918-b18a-9ed3d1bbf7b4" containerName="manager"
Jul 13 21:18:40.105108 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.105077    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead8d2af-531c-4918-b18a-9ed3d1bbf7b4" containerName="manager"
Jul 13 21:18:40.147890 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.147861    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"]
Jul 13 21:18:40.148043 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.147972    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.150527 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.150507    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zf9ww\""
Jul 13 21:18:40.150817 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.150800    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 21:18:40.151430 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.151416    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 13 21:18:40.151519 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.151422    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 21:18:40.213973 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.213949    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:40.253838 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.253814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b-maas-api-tls\") pod \"maas-api-5dcfb87784-2t785\" (UID: \"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b\") " pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.253941 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.253850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b-openshift-service-ca\") pod \"maas-api-5dcfb87784-2t785\" (UID: \"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b\") " pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.253941 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.253900    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8fg\" (UniqueName: \"kubernetes.io/projected/0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b-kube-api-access-pg8fg\") pod \"maas-api-5dcfb87784-2t785\" (UID: \"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b\") " pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.354376 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.354349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8fg\" (UniqueName: \"kubernetes.io/projected/0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b-kube-api-access-pg8fg\") pod \"maas-api-5dcfb87784-2t785\" (UID: \"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b\") " pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.354466 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.354403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b-maas-api-tls\") pod \"maas-api-5dcfb87784-2t785\" (UID: \"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b\") " pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.354466 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.354421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b-openshift-service-ca\") pod \"maas-api-5dcfb87784-2t785\" (UID: \"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b\") " pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.355052 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.355011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b-openshift-service-ca\") pod \"maas-api-5dcfb87784-2t785\" (UID: \"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b\") " pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.356797 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.356780    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b-maas-api-tls\") pod \"maas-api-5dcfb87784-2t785\" (UID: \"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b\") " pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.366974 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.366953    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8fg\" (UniqueName: \"kubernetes.io/projected/0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b-kube-api-access-pg8fg\") pod \"maas-api-5dcfb87784-2t785\" (UID: \"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b\") " pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.457724 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.457700    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:40.588336 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.588311    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"]
Jul 13 21:18:40.591277 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:18:40.591246    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf8cc9a_88eb_40b8_9dee_9f303dbacd8b.slice/crio-88c924527a1e39d40543714ab44ab2c97c6e618df1dfd26da8d74856b608deb0 WatchSource:0}: Error finding container 88c924527a1e39d40543714ab44ab2c97c6e618df1dfd26da8d74856b608deb0: Status 404 returned error can't find the container with id 88c924527a1e39d40543714ab44ab2c97c6e618df1dfd26da8d74856b608deb0
Jul 13 21:18:40.623515 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.623453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:40.717778 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.717740    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:40.717951 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.717933    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"]
Jul 13 21:18:40.720787 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.720771    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:18:40.723335 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.723315    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 13 21:18:40.731484 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.731464    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"]
Jul 13 21:18:40.758517 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.758495    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a7afd7-7a18-4b08-85ac-7787ef89fa11-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-h48dj\" (UID: \"b2a7afd7-7a18-4b08-85ac-7787ef89fa11\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:18:40.758645 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.758522    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfb4\" (UniqueName: \"kubernetes.io/projected/b2a7afd7-7a18-4b08-85ac-7787ef89fa11-kube-api-access-jqfb4\") pod \"payload-pre-processing-74584bb9f8-h48dj\" (UID: \"b2a7afd7-7a18-4b08-85ac-7787ef89fa11\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:18:40.859601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.859542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a7afd7-7a18-4b08-85ac-7787ef89fa11-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-h48dj\" (UID: \"b2a7afd7-7a18-4b08-85ac-7787ef89fa11\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:18:40.859601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.859576    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfb4\" (UniqueName: \"kubernetes.io/projected/b2a7afd7-7a18-4b08-85ac-7787ef89fa11-kube-api-access-jqfb4\") pod \"payload-pre-processing-74584bb9f8-h48dj\" (UID: \"b2a7afd7-7a18-4b08-85ac-7787ef89fa11\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:18:40.860558 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.860536    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a7afd7-7a18-4b08-85ac-7787ef89fa11-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-h48dj\" (UID: \"b2a7afd7-7a18-4b08-85ac-7787ef89fa11\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:18:40.868632 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:40.868573    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfb4\" (UniqueName: \"kubernetes.io/projected/b2a7afd7-7a18-4b08-85ac-7787ef89fa11-kube-api-access-jqfb4\") pod \"payload-pre-processing-74584bb9f8-h48dj\" (UID: \"b2a7afd7-7a18-4b08-85ac-7787ef89fa11\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:18:41.030788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:41.030764    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:18:41.091863 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:41.091827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785" event={"ID":"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b","Type":"ContainerStarted","Data":"88c924527a1e39d40543714ab44ab2c97c6e618df1dfd26da8d74856b608deb0"}
Jul 13 21:18:41.159424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:41.159337    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"]
Jul 13 21:18:41.161763 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:18:41.161735    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a7afd7_7a18_4b08_85ac_7787ef89fa11.slice/crio-2631b4458d1d6724fcfcdc72f3b3e81dda6d678de33af2b6fb18a38b51cc89ad WatchSource:0}: Error finding container 2631b4458d1d6724fcfcdc72f3b3e81dda6d678de33af2b6fb18a38b51cc89ad: Status 404 returned error can't find the container with id 2631b4458d1d6724fcfcdc72f3b3e81dda6d678de33af2b6fb18a38b51cc89ad
Jul 13 21:18:42.098525 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:42.098484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj" event={"ID":"b2a7afd7-7a18-4b08-85ac-7787ef89fa11","Type":"ContainerStarted","Data":"2631b4458d1d6724fcfcdc72f3b3e81dda6d678de33af2b6fb18a38b51cc89ad"}
Jul 13 21:18:43.103953 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:43.103912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785" event={"ID":"0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b","Type":"ContainerStarted","Data":"0495135423b746c5d207981d763baeace67c5fa91f627dae24b283fd53165de8"}
Jul 13 21:18:43.104353 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:43.103983    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:43.105351 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:43.105325    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj" event={"ID":"b2a7afd7-7a18-4b08-85ac-7787ef89fa11","Type":"ContainerStarted","Data":"7ef77f515a6f189440d533a10e4831e792e856a084edbf2a7acf9431195cc3a6"}
Jul 13 21:18:43.105446 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:43.105434    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:18:43.123311 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:43.123256    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785" podStartSLOduration=1.174378062 podStartE2EDuration="3.123238259s" totalImagesPullingTime="1.948860197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:18:43.121717694 +0000 UTC m=+569.779374725" watchObservedRunningTime="2026-07-13 21:18:43.123238259 +0000 UTC m=+569.780895266"
Jul 13 21:18:43.137951 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:43.137914    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj" podStartSLOduration=1.7202687220000001 podStartE2EDuration="3.137900248s" totalImagesPullingTime="1.417631526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:18:43.136649808 +0000 UTC m=+569.794306833" watchObservedRunningTime="2026-07-13 21:18:43.137900248 +0000 UTC m=+569.795557255"
Jul 13 21:18:49.114914 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:49.114878    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5dcfb87784-2t785"
Jul 13 21:18:50.103947 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:50.103911    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:50.199699 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:50.199668    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:18:54.111735 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:18:54.111699    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-h48dj"
Jul 13 21:19:10.381101 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:10.381035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:19:13.798439 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:13.798413    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:19:13.798901 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:13.798880    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:19:22.862655 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:22.862622    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"]
Jul 13 21:19:22.867081 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:22.867067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:22.869802 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:22.869781    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 13 21:19:22.869802 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:22.869790    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8779s\""
Jul 13 21:19:22.870848 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:22.870832    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 13 21:19:22.870927 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:22.870847    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 13 21:19:22.876779 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:22.876751    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"]
Jul 13 21:19:22.889044 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:22.889021    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:19:23.048354 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.048328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776c9e66-8c56-4622-91db-9c06ae13662f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.048518 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.048372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.048518 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.048456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.048518 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.048498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.048696 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.048520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s94sn\" (UniqueName: \"kubernetes.io/projected/776c9e66-8c56-4622-91db-9c06ae13662f-kube-api-access-s94sn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.048696 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.048576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.149220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.149152    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.149220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.149195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.149220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.149219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.149420 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.149345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s94sn\" (UniqueName: \"kubernetes.io/projected/776c9e66-8c56-4622-91db-9c06ae13662f-kube-api-access-s94sn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.149420 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.149382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.149517 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.149421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776c9e66-8c56-4622-91db-9c06ae13662f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.149634 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.149569    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.149702 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.149631    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.149804 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.149783    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.151761 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.151731    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776c9e66-8c56-4622-91db-9c06ae13662f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.152034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.152014    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776c9e66-8c56-4622-91db-9c06ae13662f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.157058 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.157035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s94sn\" (UniqueName: \"kubernetes.io/projected/776c9e66-8c56-4622-91db-9c06ae13662f-kube-api-access-s94sn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svbnb\" (UID: \"776c9e66-8c56-4622-91db-9c06ae13662f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.177039 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.177016    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:23.307494 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:23.307468    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"]
Jul 13 21:19:23.311016 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:19:23.310990    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776c9e66_8c56_4622_91db_9c06ae13662f.slice/crio-a35d14c3eddc519e7e7fce657f914595a0d94519fdf49e72b22222c3dc68a549 WatchSource:0}: Error finding container a35d14c3eddc519e7e7fce657f914595a0d94519fdf49e72b22222c3dc68a549: Status 404 returned error can't find the container with id a35d14c3eddc519e7e7fce657f914595a0d94519fdf49e72b22222c3dc68a549
Jul 13 21:19:24.272515 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:24.272479    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb" event={"ID":"776c9e66-8c56-4622-91db-9c06ae13662f","Type":"ContainerStarted","Data":"a35d14c3eddc519e7e7fce657f914595a0d94519fdf49e72b22222c3dc68a549"}
Jul 13 21:19:24.363468 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:24.363427    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:19:25.278296 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:25.278262    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb" event={"ID":"776c9e66-8c56-4622-91db-9c06ae13662f","Type":"ContainerStarted","Data":"2a4cd365517accf931d9a105376d387cc1cd17733e95a47c8eb32f21e7674d16"}
Jul 13 21:19:25.294819 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:25.294775    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb" podStartSLOduration=2.179834019 podStartE2EDuration="3.29476065s" totalImagesPullingTime="1.114926631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:19:25.293864825 +0000 UTC m=+611.951521832" watchObservedRunningTime="2026-07-13 21:19:25.29476065 +0000 UTC m=+611.952417655"
Jul 13 21:19:30.331291 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:30.331254    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:19:33.177845 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:33.177796    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:33.178355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:33.177875    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:33.190223 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:33.190195    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:33.322394 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:33.322370    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svbnb"
Jul 13 21:19:42.305125 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:42.305038    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:19:44.462349 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:44.462310    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:19:54.558439 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.558407    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"]
Jul 13 21:19:54.561779 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.561756    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.565852 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.565833    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 13 21:19:54.572153 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.572134    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"]
Jul 13 21:19:54.584917 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.584882    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:19:54.675017 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.674989    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecbc313a-2b4f-427b-aa44-79623dd0de68-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.675126 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.675026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.675126 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.675053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.675126 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.675072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kkdr\" (UniqueName: \"kubernetes.io/projected/ecbc313a-2b4f-427b-aa44-79623dd0de68-kube-api-access-2kkdr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.675312 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.675145    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.675312 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.675180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.775556 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.775529    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.775698 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.775561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kkdr\" (UniqueName: \"kubernetes.io/projected/ecbc313a-2b4f-427b-aa44-79623dd0de68-kube-api-access-2kkdr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.775698 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.775608    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.775698 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.775634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.775698 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.775667    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecbc313a-2b4f-427b-aa44-79623dd0de68-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.775898 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.775777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.776044 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.776023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.776106 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.776084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.776156 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.776118    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.777830 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.777812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecbc313a-2b4f-427b-aa44-79623dd0de68-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.778276 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.778254    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecbc313a-2b4f-427b-aa44-79623dd0de68-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.784215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.784192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kkdr\" (UniqueName: \"kubernetes.io/projected/ecbc313a-2b4f-427b-aa44-79623dd0de68-kube-api-access-2kkdr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw\" (UID: \"ecbc313a-2b4f-427b-aa44-79623dd0de68\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:54.872289 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:54.872237    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:19:55.009237 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:55.009212    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"]
Jul 13 21:19:55.011730 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:19:55.011703    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbc313a_2b4f_427b_aa44_79623dd0de68.slice/crio-6c7e4b830aa27f1b7ab74a0d6cc5df91cff7b5ba9913c3a4999031642765d465 WatchSource:0}: Error finding container 6c7e4b830aa27f1b7ab74a0d6cc5df91cff7b5ba9913c3a4999031642765d465: Status 404 returned error can't find the container with id 6c7e4b830aa27f1b7ab74a0d6cc5df91cff7b5ba9913c3a4999031642765d465
Jul 13 21:19:55.013004 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:55.012988    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:19:55.401813 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:55.401781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw" event={"ID":"ecbc313a-2b4f-427b-aa44-79623dd0de68","Type":"ContainerStarted","Data":"95b44b0d368a17374ce44a6f96795d630846d82ba39df3380663f7a492597a77"}
Jul 13 21:19:55.401973 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:55.401830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw" event={"ID":"ecbc313a-2b4f-427b-aa44-79623dd0de68","Type":"ContainerStarted","Data":"6c7e4b830aa27f1b7ab74a0d6cc5df91cff7b5ba9913c3a4999031642765d465"}
Jul 13 21:19:55.430852 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:55.430801    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw" podStartSLOduration=1.2617838670000001 podStartE2EDuration="1.430786969s" totalImagesPullingTime="169.003102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:19:55.429759609 +0000 UTC m=+642.087416616" watchObservedRunningTime="2026-07-13 21:19:55.430786969 +0000 UTC m=+642.088443975"
Jul 13 21:19:55.704005 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:19:55.703977    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:20:01.224416 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:20:01.224376    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:20:04.873105 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:20:04.873075    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:20:04.873479 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:20:04.873119    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:20:04.885443 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:20:04.885422    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:20:05.454635 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:20:05.454603    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw"
Jul 13 21:20:16.896047 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:20:16.896001    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:21:08.857110 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:21:08.857018    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:21:11.238379 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:21:11.238336    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:21:11.332336 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:21:11.332298    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:21:12.312906 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:21:12.312875    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:21:12.433913 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:21:12.433876    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:21:25.336562 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:21:25.336516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:21:25.395151 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:21:25.395127    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:22:22.746013 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:22.745973    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:22:22.845481 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:22.845455    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66ffdf856b-d9ddr"]
Jul 13 21:22:22.845668 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:22.845640    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66ffdf856b-d9ddr" podUID="748755b7-b510-4d97-9c8b-c019600831a2" containerName="manager" containerID="cri-o://674c296a7ef45040246facce00eacd210772602bc9f906dea207f76ddcd68107" gracePeriod=30
Jul 13 21:22:23.027579 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:23.027551    2564 generic.go:356] "Generic (PLEG): container finished" podID="748755b7-b510-4d97-9c8b-c019600831a2" containerID="674c296a7ef45040246facce00eacd210772602bc9f906dea207f76ddcd68107" exitCode=0
Jul 13 21:22:23.027579 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:23.027606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66ffdf856b-d9ddr" event={"ID":"748755b7-b510-4d97-9c8b-c019600831a2","Type":"ContainerDied","Data":"674c296a7ef45040246facce00eacd210772602bc9f906dea207f76ddcd68107"}
Jul 13 21:22:23.081319 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:23.081293    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:22:23.160564 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:23.160541    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/748755b7-b510-4d97-9c8b-c019600831a2-webhook-cert\") pod \"748755b7-b510-4d97-9c8b-c019600831a2\" (UID: \"748755b7-b510-4d97-9c8b-c019600831a2\") "
Jul 13 21:22:23.160705 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:23.160615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct6sx\" (UniqueName: \"kubernetes.io/projected/748755b7-b510-4d97-9c8b-c019600831a2-kube-api-access-ct6sx\") pod \"748755b7-b510-4d97-9c8b-c019600831a2\" (UID: \"748755b7-b510-4d97-9c8b-c019600831a2\") "
Jul 13 21:22:23.162638 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:23.162607    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/748755b7-b510-4d97-9c8b-c019600831a2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "748755b7-b510-4d97-9c8b-c019600831a2" (UID: "748755b7-b510-4d97-9c8b-c019600831a2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:22:23.162741 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:23.162644    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748755b7-b510-4d97-9c8b-c019600831a2-kube-api-access-ct6sx" (OuterVolumeSpecName: "kube-api-access-ct6sx") pod "748755b7-b510-4d97-9c8b-c019600831a2" (UID: "748755b7-b510-4d97-9c8b-c019600831a2"). InnerVolumeSpecName "kube-api-access-ct6sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:22:23.262126 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:23.262055    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/748755b7-b510-4d97-9c8b-c019600831a2-webhook-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:22:23.262126 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:23.262079    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct6sx\" (UniqueName: \"kubernetes.io/projected/748755b7-b510-4d97-9c8b-c019600831a2-kube-api-access-ct6sx\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:22:24.035393 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.035365    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66ffdf856b-d9ddr"
Jul 13 21:22:24.035393 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.035370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66ffdf856b-d9ddr" event={"ID":"748755b7-b510-4d97-9c8b-c019600831a2","Type":"ContainerDied","Data":"833e45fec181c273d4d064bce1a2a09e238a07dfcb9e07268ef05a688c7b97d2"}
Jul 13 21:22:24.035857 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.035432    2564 scope.go:117] "RemoveContainer" containerID="674c296a7ef45040246facce00eacd210772602bc9f906dea207f76ddcd68107"
Jul 13 21:22:24.050930 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.050906    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66ffdf856b-d9ddr"]
Jul 13 21:22:24.056349 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.056328    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66ffdf856b-d9ddr"]
Jul 13 21:22:24.672369 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.672344    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:22:24.776576 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.776549    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66ffdf856b-zh9sw"]
Jul 13 21:22:24.776862 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.776848    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="748755b7-b510-4d97-9c8b-c019600831a2" containerName="manager"
Jul 13 21:22:24.776913 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.776864    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="748755b7-b510-4d97-9c8b-c019600831a2" containerName="manager"
Jul 13 21:22:24.776946 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.776932    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="748755b7-b510-4d97-9c8b-c019600831a2" containerName="manager"
Jul 13 21:22:24.780852 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.780838    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:22:24.784111 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.784087    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 13 21:22:24.784111 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.784109    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 13 21:22:24.784111 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.784110    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2k6tc\""
Jul 13 21:22:24.786660 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.786575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66ffdf856b-zh9sw"]
Jul 13 21:22:24.875355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.875321    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4c13ca7-df5e-452f-829e-2575b90a05e1-webhook-cert\") pod \"maas-controller-66ffdf856b-zh9sw\" (UID: \"b4c13ca7-df5e-452f-829e-2575b90a05e1\") " pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:22:24.875463 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.875364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-967wt\" (UniqueName: \"kubernetes.io/projected/b4c13ca7-df5e-452f-829e-2575b90a05e1-kube-api-access-967wt\") pod \"maas-controller-66ffdf856b-zh9sw\" (UID: \"b4c13ca7-df5e-452f-829e-2575b90a05e1\") " pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:22:24.976778 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.976706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4c13ca7-df5e-452f-829e-2575b90a05e1-webhook-cert\") pod \"maas-controller-66ffdf856b-zh9sw\" (UID: \"b4c13ca7-df5e-452f-829e-2575b90a05e1\") " pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:22:24.976778 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.976755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-967wt\" (UniqueName: \"kubernetes.io/projected/b4c13ca7-df5e-452f-829e-2575b90a05e1-kube-api-access-967wt\") pod \"maas-controller-66ffdf856b-zh9sw\" (UID: \"b4c13ca7-df5e-452f-829e-2575b90a05e1\") " pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:22:24.979204 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.979180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4c13ca7-df5e-452f-829e-2575b90a05e1-webhook-cert\") pod \"maas-controller-66ffdf856b-zh9sw\" (UID: \"b4c13ca7-df5e-452f-829e-2575b90a05e1\") " pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:22:24.984638 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:24.984617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-967wt\" (UniqueName: \"kubernetes.io/projected/b4c13ca7-df5e-452f-829e-2575b90a05e1-kube-api-access-967wt\") pod \"maas-controller-66ffdf856b-zh9sw\" (UID: \"b4c13ca7-df5e-452f-829e-2575b90a05e1\") " pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:22:25.092263 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:25.092240    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:22:25.222881 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:25.222850    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66ffdf856b-zh9sw"]
Jul 13 21:22:25.225773 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:22:25.225734    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c13ca7_df5e_452f_829e_2575b90a05e1.slice/crio-e2c6cf1b9730b4f945be1509cc6baa60f646bcfa02f876900d943815bfde0e38 WatchSource:0}: Error finding container e2c6cf1b9730b4f945be1509cc6baa60f646bcfa02f876900d943815bfde0e38: Status 404 returned error can't find the container with id e2c6cf1b9730b4f945be1509cc6baa60f646bcfa02f876900d943815bfde0e38
Jul 13 21:22:25.821425 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:25.821397    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="748755b7-b510-4d97-9c8b-c019600831a2" path="/var/lib/kubelet/pods/748755b7-b510-4d97-9c8b-c019600831a2/volumes"
Jul 13 21:22:26.045558 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:26.045525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66ffdf856b-zh9sw" event={"ID":"b4c13ca7-df5e-452f-829e-2575b90a05e1","Type":"ContainerStarted","Data":"7708a12e8bb100369c268b3f82bf2f1a74d6e1dc4fd0937dbf95f41012b15cc5"}
Jul 13 21:22:26.045736 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:26.045578    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66ffdf856b-zh9sw" event={"ID":"b4c13ca7-df5e-452f-829e-2575b90a05e1","Type":"ContainerStarted","Data":"e2c6cf1b9730b4f945be1509cc6baa60f646bcfa02f876900d943815bfde0e38"}
Jul 13 21:22:26.045736 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:26.045624    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:22:26.061114 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:26.061068    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66ffdf856b-zh9sw" podStartSLOduration=1.583795194 podStartE2EDuration="2.061051137s" totalImagesPullingTime="477.255943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:22:26.060066582 +0000 UTC m=+792.717723600" watchObservedRunningTime="2026-07-13 21:22:26.061051137 +0000 UTC m=+792.718708155"
Jul 13 21:22:37.051924 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:22:37.051856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66ffdf856b-zh9sw"
Jul 13 21:23:05.044320 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:23:05.044270    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:23:05.178959 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:23:05.178917    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:23:05.278863 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:23:05.278842    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:23:10.974553 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:23:10.974508    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:23:11.074082 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:23:11.074052    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:23:11.174392 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:23:11.174355    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:23:11.313729 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:23:11.313702    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:23:17.093170 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:23:17.093114    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:23:17.198077 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:23:17.198047    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:24:13.827666 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:24:13.827600    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:24:13.829718 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:24:13.829697    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:24:23.081528 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:24:23.081488    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:24:23.178303 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:24:23.178277    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:24:44.942041 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:24:44.942002    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:25:24.706232 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:25:24.706190    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:25:24.761161 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:25:24.761131    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:25:58.497476 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:25:58.497367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:13.849467 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:29:13.849440    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:29:13.852393 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:29:13.852371    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:30:00.145312 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:00.145273    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc"]
Jul 13 21:30:00.148449 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:00.148432    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc"
Jul 13 21:30:00.150889 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:00.150868    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:30:00.157783 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:00.157765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc"
Jul 13 21:30:00.163461 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:00.163433    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc"]
Jul 13 21:30:00.294571 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:00.294540    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc"]
Jul 13 21:30:00.297783 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:30:00.297759    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc7b126_cb9b_4218_8450_eea9e3cdf6a5.slice/crio-c9e1539751a55d31a1872b86f07b6f0a7421174dffada1a1b45443a47b358120 WatchSource:0}: Error finding container c9e1539751a55d31a1872b86f07b6f0a7421174dffada1a1b45443a47b358120: Status 404 returned error can't find the container with id c9e1539751a55d31a1872b86f07b6f0a7421174dffada1a1b45443a47b358120
Jul 13 21:30:00.302773 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:00.302752    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:30:00.926936 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:00.926904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc" event={"ID":"7bc7b126-cb9b-4218-8450-eea9e3cdf6a5","Type":"ContainerStarted","Data":"c9e1539751a55d31a1872b86f07b6f0a7421174dffada1a1b45443a47b358120"}
Jul 13 21:30:02.938229 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:02.938194    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc" event={"ID":"7bc7b126-cb9b-4218-8450-eea9e3cdf6a5","Type":"ContainerStarted","Data":"743b160e33f3b6fafddae6aee65d85f377e9b1266d9523c681261885f17cd11f"}
Jul 13 21:30:02.953243 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:02.953196    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc" podStartSLOduration=0.931667376 podStartE2EDuration="2.953178981s" totalImagesPullingTime="2.021511605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:30:02.951676957 +0000 UTC m=+1249.609333988" watchObservedRunningTime="2026-07-13 21:30:02.953178981 +0000 UTC m=+1249.610835988"
Jul 13 21:30:03.944241 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:03.944204    2564 generic.go:356] "Generic (PLEG): container finished" podID="7bc7b126-cb9b-4218-8450-eea9e3cdf6a5" containerID="743b160e33f3b6fafddae6aee65d85f377e9b1266d9523c681261885f17cd11f" exitCode=0
Jul 13 21:30:03.944241 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:03.944251    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc" event={"ID":"7bc7b126-cb9b-4218-8450-eea9e3cdf6a5","Type":"ContainerDied","Data":"743b160e33f3b6fafddae6aee65d85f377e9b1266d9523c681261885f17cd11f"}
Jul 13 21:30:05.080762 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:05.080740    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc"
Jul 13 21:30:05.953403 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:05.953379    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc"
Jul 13 21:30:05.953663 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:05.953377    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732970-dqbfc" event={"ID":"7bc7b126-cb9b-4218-8450-eea9e3cdf6a5","Type":"ContainerDied","Data":"c9e1539751a55d31a1872b86f07b6f0a7421174dffada1a1b45443a47b358120"}
Jul 13 21:30:05.953663 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:30:05.953519    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9e1539751a55d31a1872b86f07b6f0a7421174dffada1a1b45443a47b358120"
Jul 13 21:31:10.014241 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:31:10.014204    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:31:41.753083 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:31:41.753045    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:01.476886 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:01.476850    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:07.984606 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:07.984547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:13.870993 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:13.870966    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:34:13.874847 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:13.874824    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:34:16.238881 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:16.238845    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:21.294058 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:21.294016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:24.189266 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:24.189228    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:24.287372 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:24.287338    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:24.629032 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:24.628992    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:25.347520 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:25.347482    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:30.503619 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.503564    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:30.571536 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.571509    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:30.665609 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.665559    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"]
Jul 13 21:34:30.665948 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.665931    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc7b126-cb9b-4218-8450-eea9e3cdf6a5" containerName="cleanup"
Jul 13 21:34:30.666033 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.665951    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc7b126-cb9b-4218-8450-eea9e3cdf6a5" containerName="cleanup"
Jul 13 21:34:30.666090 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.666051    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc7b126-cb9b-4218-8450-eea9e3cdf6a5" containerName="cleanup"
Jul 13 21:34:30.668879 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.668855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.671196 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.671177    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6f1f77b0-openshift-default-dockercfg-pzjbk\""
Jul 13 21:34:30.689150 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.689127    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"]
Jul 13 21:34:30.697538 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.697509    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:30.754060 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.753987    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:30.755985 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.755961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-podinfo\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.756069 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.756005    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-certs\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.756069 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.756063    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-token\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.756142 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.756086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-envoy\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.756176 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.756159    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl72g\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-kube-api-access-jl72g\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.756209 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.756188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-data\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.756241 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.756208    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-socket\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.756241 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.756232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0bdc11-75f8-4bda-889b-0339d0248f92-istiod-ca-cert\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.756307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.756274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-credential-socket\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.856831 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.856801    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-credential-socket\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.856941 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.856837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-podinfo\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.856941 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.856865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-certs\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.856941 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.856892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-token\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.856941 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.856910    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-envoy\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.856941 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.856932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl72g\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-kube-api-access-jl72g\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.857182 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.856950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-data\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.857182 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.856968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-socket\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.857182 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.856985    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0bdc11-75f8-4bda-889b-0339d0248f92-istiod-ca-cert\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.857333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.857253    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-credential-socket\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.857430 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.857381    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-certs\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.857430 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.857400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-data\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.857542 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.857476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-socket\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.857798 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.857743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0bdc11-75f8-4bda-889b-0339d0248f92-istiod-ca-cert\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.859480 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.859460    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-envoy\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.859718 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.859700    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-podinfo\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.864881 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.864850    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-token\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.865042 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.865022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl72g\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-kube-api-access-jl72g\") pod \"e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") " pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:30.980181 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:30.980155    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:31.111002 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:31.110974    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"]
Jul 13 21:34:31.113362 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:34:31.113335    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0bdc11_75f8_4bda_889b_0339d0248f92.slice/crio-6e15ab4e39b95cd7d0ac0fb4b69fca8939a246eca729a0d9ab5c4f4f3a47069f WatchSource:0}: Error finding container 6e15ab4e39b95cd7d0ac0fb4b69fca8939a246eca729a0d9ab5c4f4f3a47069f: Status 404 returned error can't find the container with id 6e15ab4e39b95cd7d0ac0fb4b69fca8939a246eca729a0d9ab5c4f4f3a47069f
Jul 13 21:34:31.538887 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:31.538828    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:31.638234 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:31.638205    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:31.805182 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:31.805082    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:31.907284 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:31.907228    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:32.032864 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.032824    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"]
Jul 13 21:34:32.036365 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.036344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.038746 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.038721    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6f1f77b0-serving-cert\""
Jul 13 21:34:32.046116 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.046089    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"]
Jul 13 21:34:32.067823 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.067733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:32.089124 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.089090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" event={"ID":"cc0bdc11-75f8-4bda-889b-0339d0248f92","Type":"ContainerStarted","Data":"6e15ab4e39b95cd7d0ac0fb4b69fca8939a246eca729a0d9ab5c4f4f3a47069f"}
Jul 13 21:34:32.167775 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.167746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vx7f\" (UniqueName: \"kubernetes.io/projected/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-kube-api-access-6vx7f\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.167917 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.167789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-openshift-service-ca\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.167988 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.167948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-maas-api-tls\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.269344 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.269306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-openshift-service-ca\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.269492 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.269398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-maas-api-tls\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.269492 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.269475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vx7f\" (UniqueName: \"kubernetes.io/projected/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-kube-api-access-6vx7f\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.269660 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:34:32.269561    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-serving-cert: secret "maas-api-e2e-ait-6f1f77b0-serving-cert" not found
Jul 13 21:34:32.269734 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:34:32.269660    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-maas-api-tls podName:33da81a9-bb0a-4cb2-9b91-e24a94608fe9 nodeName:}" failed. No retries permitted until 2026-07-13 21:34:32.769630563 +0000 UTC m=+1519.427287548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-maas-api-tls") pod "maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2" (UID: "33da81a9-bb0a-4cb2-9b91-e24a94608fe9") : secret "maas-api-e2e-ait-6f1f77b0-serving-cert" not found
Jul 13 21:34:32.270222 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.270188    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-openshift-service-ca\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.278172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.278146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vx7f\" (UniqueName: \"kubernetes.io/projected/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-kube-api-access-6vx7f\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.331685 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.331507    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:32.498970 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.498930    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:32.600954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.600862    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:32.773406 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.773374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-maas-api-tls\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.776257 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.776235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-maas-api-tls\") pod \"maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:32.950970 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:32.950886    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:33.751433 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:33.751397    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:34:33.751722 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:33.751499    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:34:33.751722 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:33.751526    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:34:33.813272 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:33.813224    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"]
Jul 13 21:34:33.815358 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:34:33.815329    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33da81a9_bb0a_4cb2_9b91_e24a94608fe9.slice/crio-a738b844af867c006e6c00b1aa34ad88993de7262f170e6016099f99d0efab77 WatchSource:0}: Error finding container a738b844af867c006e6c00b1aa34ad88993de7262f170e6016099f99d0efab77: Status 404 returned error can't find the container with id a738b844af867c006e6c00b1aa34ad88993de7262f170e6016099f99d0efab77
Jul 13 21:34:34.100188 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:34.100149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2" event={"ID":"33da81a9-bb0a-4cb2-9b91-e24a94608fe9","Type":"ContainerStarted","Data":"a738b844af867c006e6c00b1aa34ad88993de7262f170e6016099f99d0efab77"}
Jul 13 21:34:34.101622 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:34.101574    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" event={"ID":"cc0bdc11-75f8-4bda-889b-0339d0248f92","Type":"ContainerStarted","Data":"0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94"}
Jul 13 21:34:34.125919 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:34.125877    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" podStartSLOduration=1.49004117 podStartE2EDuration="4.125864238s" totalImagesPullingTime="2.635823068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:34.125136019 +0000 UTC m=+1520.782793051" watchObservedRunningTime="2026-07-13 21:34:34.125864238 +0000 UTC m=+1520.783521243"
Jul 13 21:34:34.665972 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:34.665917    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:34.981274 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:34.981091    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:34.994405 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:34.985724    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:35.106922 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:35.106898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2" event={"ID":"33da81a9-bb0a-4cb2-9b91-e24a94608fe9","Type":"ContainerStarted","Data":"e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6"}
Jul 13 21:34:35.107135 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:35.107117    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:35.107211 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:35.107164    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:35.107810 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:35.107793    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:35.122288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:35.122251    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2" podStartSLOduration=2.584642135 podStartE2EDuration="3.122239224s" totalImagesPullingTime="537.597089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:35.121815541 +0000 UTC m=+1521.779472547" watchObservedRunningTime="2026-07-13 21:34:35.122239224 +0000 UTC m=+1521.779896229"
Jul 13 21:34:36.888129 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:36.888094    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"]
Jul 13 21:34:36.891313 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:36.891293    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"
Jul 13 21:34:36.893650 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:36.893620    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:34:36.898538 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:36.898515    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"]
Jul 13 21:34:37.009425 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:37.009392    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c9f32e-3b73-4c4c-83ee-953cf310435b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f\" (UID: \"a1c9f32e-3b73-4c4c-83ee-953cf310435b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"
Jul 13 21:34:37.110671 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:37.110640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c9f32e-3b73-4c4c-83ee-953cf310435b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f\" (UID: \"a1c9f32e-3b73-4c4c-83ee-953cf310435b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"
Jul 13 21:34:37.111244 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:37.111216    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c9f32e-3b73-4c4c-83ee-953cf310435b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f\" (UID: \"a1c9f32e-3b73-4c4c-83ee-953cf310435b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"
Jul 13 21:34:37.203012 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:37.202929    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"
Jul 13 21:34:37.368622 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:37.368492    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"]
Jul 13 21:34:37.372841 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:34:37.372810    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c9f32e_3b73_4c4c_83ee_953cf310435b.slice/crio-178305ba2b653399026c36b374002171fb32e85a6211de5e4e0ec927db58b5a1 WatchSource:0}: Error finding container 178305ba2b653399026c36b374002171fb32e85a6211de5e4e0ec927db58b5a1: Status 404 returned error can't find the container with id 178305ba2b653399026c36b374002171fb32e85a6211de5e4e0ec927db58b5a1
Jul 13 21:34:38.121461 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:38.121429    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" event={"ID":"a1c9f32e-3b73-4c4c-83ee-953cf310435b","Type":"ContainerStarted","Data":"a5ea1bd4e54871d85b41409034765049a380c626ba67512454fe09081a85b973"}
Jul 13 21:34:38.121959 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:38.121492    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" event={"ID":"a1c9f32e-3b73-4c4c-83ee-953cf310435b","Type":"ContainerStarted","Data":"178305ba2b653399026c36b374002171fb32e85a6211de5e4e0ec927db58b5a1"}
Jul 13 21:34:38.136809 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:38.136764    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" podStartSLOduration=2.13674919 podStartE2EDuration="2.13674919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:38.134889535 +0000 UTC m=+1524.792546543" watchObservedRunningTime="2026-07-13 21:34:38.13674919 +0000 UTC m=+1524.794406196"
Jul 13 21:34:39.127718 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:39.127685    2564 generic.go:356] "Generic (PLEG): container finished" podID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerID="a5ea1bd4e54871d85b41409034765049a380c626ba67512454fe09081a85b973" exitCode=7
Jul 13 21:34:39.128093 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:39.127768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" event={"ID":"a1c9f32e-3b73-4c4c-83ee-953cf310435b","Type":"ContainerDied","Data":"a5ea1bd4e54871d85b41409034765049a380c626ba67512454fe09081a85b973"}
Jul 13 21:34:39.128093 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:39.127929    2564 scope.go:117] "RemoveContainer" containerID="a5ea1bd4e54871d85b41409034765049a380c626ba67512454fe09081a85b973"
Jul 13 21:34:40.135454 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:40.135424    2564 generic.go:356] "Generic (PLEG): container finished" podID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerID="87baab5cee7ffb4518bf4df0c5921663c3a5a975e3385632d98563d35b02a289" exitCode=7
Jul 13 21:34:40.135825 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:40.135492    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" event={"ID":"a1c9f32e-3b73-4c4c-83ee-953cf310435b","Type":"ContainerDied","Data":"87baab5cee7ffb4518bf4df0c5921663c3a5a975e3385632d98563d35b02a289"}
Jul 13 21:34:40.135825 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:40.135560    2564 scope.go:117] "RemoveContainer" containerID="a5ea1bd4e54871d85b41409034765049a380c626ba67512454fe09081a85b973"
Jul 13 21:34:40.135825 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:40.135776    2564 scope.go:117] "RemoveContainer" containerID="87baab5cee7ffb4518bf4df0c5921663c3a5a975e3385632d98563d35b02a289"
Jul 13 21:34:40.136022 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:34:40.136003    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f_odh-ai-gateway-infra(a1c9f32e-3b73-4c4c-83ee-953cf310435b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b"
Jul 13 21:34:41.116842 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:41.116812    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:34:41.142476 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:41.142452    2564 scope.go:117] "RemoveContainer" containerID="87baab5cee7ffb4518bf4df0c5921663c3a5a975e3385632d98563d35b02a289"
Jul 13 21:34:41.143201 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:34:41.143165    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f_odh-ai-gateway-infra(a1c9f32e-3b73-4c4c-83ee-953cf310435b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b"
Jul 13 21:34:43.152765 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:43.152733    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerID="0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94" exitCode=0
Jul 13 21:34:43.153088 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:43.152800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" event={"ID":"cc0bdc11-75f8-4bda-889b-0339d0248f92","Type":"ContainerDied","Data":"0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94"}
Jul 13 21:34:43.153213 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:43.153196    2564 scope.go:117] "RemoveContainer" containerID="0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94"
Jul 13 21:34:43.155062 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:43.155040    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:34:43.155123 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:43.155109    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:34:43.155165 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:43.155145    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:34:44.162761 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:44.162718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" event={"ID":"cc0bdc11-75f8-4bda-889b-0339d0248f92","Type":"ContainerStarted","Data":"698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c"}
Jul 13 21:34:44.980764 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:44.980726    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:44.980939 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:44.980896    2564 patch_prober.go:28] interesting pod/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Jul 13 21:34:44.980989 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:44.980944    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Jul 13 21:34:45.981036 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:45.980997    2564 patch_prober.go:28] interesting pod/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Jul 13 21:34:45.981441 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:45.981077    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Jul 13 21:34:46.980747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:46.980712    2564 patch_prober.go:28] interesting pod/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Jul 13 21:34:46.980921 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:46.980789    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Jul 13 21:34:47.980994 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:47.980955    2564 patch_prober.go:28] interesting pod/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Jul 13 21:34:47.981390 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:47.981007    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Jul 13 21:34:48.985014 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:48.984983    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:48.985447 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:48.985425    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:48.986113 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:48.986096    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:34:53.822753 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:53.822722    2564 scope.go:117] "RemoveContainer" containerID="87baab5cee7ffb4518bf4df0c5921663c3a5a975e3385632d98563d35b02a289"
Jul 13 21:34:54.208669 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:54.208641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" event={"ID":"a1c9f32e-3b73-4c4c-83ee-953cf310435b","Type":"ContainerStarted","Data":"0428dd2760f7fd0e76e116e58181a3bbd507441ff2fa451f70a1d40ee06a400e"}
Jul 13 21:34:55.214443 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:55.214408    2564 generic.go:356] "Generic (PLEG): container finished" podID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerID="0428dd2760f7fd0e76e116e58181a3bbd507441ff2fa451f70a1d40ee06a400e" exitCode=0
Jul 13 21:34:55.214443 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:55.214443    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" event={"ID":"a1c9f32e-3b73-4c4c-83ee-953cf310435b","Type":"ContainerDied","Data":"0428dd2760f7fd0e76e116e58181a3bbd507441ff2fa451f70a1d40ee06a400e"}
Jul 13 21:34:55.214915 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:55.214488    2564 scope.go:117] "RemoveContainer" containerID="87baab5cee7ffb4518bf4df0c5921663c3a5a975e3385632d98563d35b02a289"
Jul 13 21:34:55.236062 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:55.236036    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"]
Jul 13 21:34:56.351233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:56.351202    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"
Jul 13 21:34:56.473710 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:56.473649    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c9f32e-3b73-4c4c-83ee-953cf310435b-maas-api-service-ca\") pod \"a1c9f32e-3b73-4c4c-83ee-953cf310435b\" (UID: \"a1c9f32e-3b73-4c4c-83ee-953cf310435b\") "
Jul 13 21:34:56.473966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:56.473945    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c9f32e-3b73-4c4c-83ee-953cf310435b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1c9f32e-3b73-4c4c-83ee-953cf310435b" (UID: "a1c9f32e-3b73-4c4c-83ee-953cf310435b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:34:56.574618 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:56.574570    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c9f32e-3b73-4c4c-83ee-953cf310435b-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:34:57.225216 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:57.225185    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"
Jul 13 21:34:57.225216 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:57.225191    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f" event={"ID":"a1c9f32e-3b73-4c4c-83ee-953cf310435b","Type":"ContainerDied","Data":"178305ba2b653399026c36b374002171fb32e85a6211de5e4e0ec927db58b5a1"}
Jul 13 21:34:57.225417 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:57.225238    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="178305ba2b653399026c36b374002171fb32e85a6211de5e4e0ec927db58b5a1"
Jul 13 21:34:57.252401 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:57.252377    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"]
Jul 13 21:34:57.254048 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:57.254027    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-czq5f"]
Jul 13 21:34:57.826426 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:34:57.821766    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" path="/var/lib/kubelet/pods/a1c9f32e-3b73-4c4c-83ee-953cf310435b/volumes"
Jul 13 21:35:36.936217 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.936178    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"]
Jul 13 21:35:36.936735 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.936625    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerName="revoke-keys"
Jul 13 21:35:36.936735 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.936642    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerName="revoke-keys"
Jul 13 21:35:36.936735 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.936674    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerName="revoke-keys"
Jul 13 21:35:36.936735 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.936683    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerName="revoke-keys"
Jul 13 21:35:36.936948 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.936770    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerName="revoke-keys"
Jul 13 21:35:36.936948 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.936783    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerName="revoke-keys"
Jul 13 21:35:36.936948 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.936791    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerName="revoke-keys"
Jul 13 21:35:36.939837 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.939817    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"
Jul 13 21:35:36.942477 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.942457    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:35:36.946683 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.946663    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"]
Jul 13 21:35:36.967237 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:36.967215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7421761-f854-423e-a1c9-04796ac4821e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7\" (UID: \"f7421761-f854-423e-a1c9-04796ac4821e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"
Jul 13 21:35:37.068000 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:37.067972    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7421761-f854-423e-a1c9-04796ac4821e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7\" (UID: \"f7421761-f854-423e-a1c9-04796ac4821e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"
Jul 13 21:35:37.068546 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:37.068527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7421761-f854-423e-a1c9-04796ac4821e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7\" (UID: \"f7421761-f854-423e-a1c9-04796ac4821e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"
Jul 13 21:35:37.252829 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:37.252803    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"
Jul 13 21:35:37.380433 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:37.380406    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"]
Jul 13 21:35:37.383343 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:35:37.383315    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7421761_f854_423e_a1c9_04796ac4821e.slice/crio-adc805201440e282263c7b41b9fc3f2da87815c87fc4f6996a361952448998b5 WatchSource:0}: Error finding container adc805201440e282263c7b41b9fc3f2da87815c87fc4f6996a361952448998b5: Status 404 returned error can't find the container with id adc805201440e282263c7b41b9fc3f2da87815c87fc4f6996a361952448998b5
Jul 13 21:35:37.414065 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:37.414036    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7" event={"ID":"f7421761-f854-423e-a1c9-04796ac4821e","Type":"ContainerStarted","Data":"adc805201440e282263c7b41b9fc3f2da87815c87fc4f6996a361952448998b5"}
Jul 13 21:35:38.423807 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:38.423722    2564 generic.go:356] "Generic (PLEG): container finished" podID="f7421761-f854-423e-a1c9-04796ac4821e" containerID="46925f80bd6e1b7bc22598b94541667ece09ccbce9ef94af5ef8ad1daa51a8fd" exitCode=0
Jul 13 21:35:38.424194 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:38.423804    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7" event={"ID":"f7421761-f854-423e-a1c9-04796ac4821e","Type":"ContainerDied","Data":"46925f80bd6e1b7bc22598b94541667ece09ccbce9ef94af5ef8ad1daa51a8fd"}
Jul 13 21:35:39.557968 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:39.557937    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"
Jul 13 21:35:39.585381 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:39.585360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7421761-f854-423e-a1c9-04796ac4821e-maas-api-service-ca\") pod \"f7421761-f854-423e-a1c9-04796ac4821e\" (UID: \"f7421761-f854-423e-a1c9-04796ac4821e\") "
Jul 13 21:35:39.585723 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:39.585696    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7421761-f854-423e-a1c9-04796ac4821e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7421761-f854-423e-a1c9-04796ac4821e" (UID: "f7421761-f854-423e-a1c9-04796ac4821e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:35:39.686465 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:39.686392    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7421761-f854-423e-a1c9-04796ac4821e-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:35:40.433726 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:40.433691    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7" event={"ID":"f7421761-f854-423e-a1c9-04796ac4821e","Type":"ContainerDied","Data":"adc805201440e282263c7b41b9fc3f2da87815c87fc4f6996a361952448998b5"}
Jul 13 21:35:40.433726 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:40.433714    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7"
Jul 13 21:35:40.433944 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:40.433751    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc805201440e282263c7b41b9fc3f2da87815c87fc4f6996a361952448998b5"
Jul 13 21:35:47.489682 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.489640    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"]
Jul 13 21:35:47.490392 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.490361    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2" podUID="33da81a9-bb0a-4cb2-9b91-e24a94608fe9" containerName="maas-api" containerID="cri-o://e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6" gracePeriod=30
Jul 13 21:35:47.519878 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.519853    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:47.734777 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.734754    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:35:47.849191 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.849121    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vx7f\" (UniqueName: \"kubernetes.io/projected/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-kube-api-access-6vx7f\") pod \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") "
Jul 13 21:35:47.849191 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.849171    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-openshift-service-ca\") pod \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") "
Jul 13 21:35:47.849360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.849207    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-maas-api-tls\") pod \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\" (UID: \"33da81a9-bb0a-4cb2-9b91-e24a94608fe9\") "
Jul 13 21:35:47.849507 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.849482    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33da81a9-bb0a-4cb2-9b91-e24a94608fe9" (UID: "33da81a9-bb0a-4cb2-9b91-e24a94608fe9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:35:47.851452 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.851428    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-kube-api-access-6vx7f" (OuterVolumeSpecName: "kube-api-access-6vx7f") pod "33da81a9-bb0a-4cb2-9b91-e24a94608fe9" (UID: "33da81a9-bb0a-4cb2-9b91-e24a94608fe9"). InnerVolumeSpecName "kube-api-access-6vx7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:35:47.851538 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.851483    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33da81a9-bb0a-4cb2-9b91-e24a94608fe9" (UID: "33da81a9-bb0a-4cb2-9b91-e24a94608fe9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:47.950708 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.950672    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:35:47.950708 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.950701    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vx7f\" (UniqueName: \"kubernetes.io/projected/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-kube-api-access-6vx7f\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:35:47.950708 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:47.950715    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33da81a9-bb0a-4cb2-9b91-e24a94608fe9-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.468453 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:48.468424    2564 generic.go:356] "Generic (PLEG): container finished" podID="33da81a9-bb0a-4cb2-9b91-e24a94608fe9" containerID="e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6" exitCode=0
Jul 13 21:35:48.468618 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:48.468470    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"
Jul 13 21:35:48.468618 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:48.468506    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2" event={"ID":"33da81a9-bb0a-4cb2-9b91-e24a94608fe9","Type":"ContainerDied","Data":"e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6"}
Jul 13 21:35:48.468618 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:48.468559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2" event={"ID":"33da81a9-bb0a-4cb2-9b91-e24a94608fe9","Type":"ContainerDied","Data":"a738b844af867c006e6c00b1aa34ad88993de7262f170e6016099f99d0efab77"}
Jul 13 21:35:48.468618 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:48.468606    2564 scope.go:117] "RemoveContainer" containerID="e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6"
Jul 13 21:35:48.477881 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:48.477865    2564 scope.go:117] "RemoveContainer" containerID="e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6"
Jul 13 21:35:48.478139 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:35:48.478120    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6\": container with ID starting with e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6 not found: ID does not exist" containerID="e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6"
Jul 13 21:35:48.478182 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:48.478148    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6"} err="failed to get container status \"e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6\": rpc error: code = NotFound desc = could not find container \"e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6\": container with ID starting with e1abed8bc880adc016818d205e3db4ae454490c2c98b22c00b0211a2dec8f8e6 not found: ID does not exist"
Jul 13 21:35:48.488302 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:48.488283    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"]
Jul 13 21:35:48.491982 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:48.491965    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f1f77b0-6b759595b8-28bv2"]
Jul 13 21:35:49.821650 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:49.821609    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33da81a9-bb0a-4cb2-9b91-e24a94608fe9" path="/var/lib/kubelet/pods/33da81a9-bb0a-4cb2-9b91-e24a94608fe9/volumes"
Jul 13 21:35:52.017729 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:52.017684    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:57.859007 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:57.858970    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:57.958977 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:35:57.958938    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:02.178434 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.178400    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"]
Jul 13 21:36:02.178998 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.178615    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy" containerID="cri-o://698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c" gracePeriod=30
Jul 13 21:36:02.200509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.200474    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:02.573544 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.573508    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:02.675561 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.675527    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:02.717577 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.717547    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"]
Jul 13 21:36:02.718045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.718022    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7421761-f854-423e-a1c9-04796ac4821e" containerName="revoke-keys"
Jul 13 21:36:02.718136 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.718050    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7421761-f854-423e-a1c9-04796ac4821e" containerName="revoke-keys"
Jul 13 21:36:02.718136 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.718069    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerName="revoke-keys"
Jul 13 21:36:02.718136 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.718077    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c9f32e-3b73-4c4c-83ee-953cf310435b" containerName="revoke-keys"
Jul 13 21:36:02.718136 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.718103    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33da81a9-bb0a-4cb2-9b91-e24a94608fe9" containerName="maas-api"
Jul 13 21:36:02.718136 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.718112    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="33da81a9-bb0a-4cb2-9b91-e24a94608fe9" containerName="maas-api"
Jul 13 21:36:02.718397 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.718198    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="33da81a9-bb0a-4cb2-9b91-e24a94608fe9" containerName="maas-api"
Jul 13 21:36:02.718397 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.718212    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7421761-f854-423e-a1c9-04796ac4821e" containerName="revoke-keys"
Jul 13 21:36:02.722920 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.722900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.725311 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.725291    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f4738480-gw-openshift-default-dockercfg-gwd2g\""
Jul 13 21:36:02.735727 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.735704    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"]
Jul 13 21:36:02.761803 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.761781    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:02.858831 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.858743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-data\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.858831 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.858792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-podinfo\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.859045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.858867    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-token\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.859045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.858916    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8925g\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-kube-api-access-8925g\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.859045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.858957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-socket\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.859045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.858980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-certs\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.859045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.859001    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-envoy\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.859045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.859034    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istiod-ca-cert\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.859253 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.859074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-credential-socket\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.959879 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.959848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-podinfo\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.959879 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.959879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-token\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960054 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.959898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8925g\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-kube-api-access-8925g\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960054 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-socket\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960152 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-certs\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960152 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960085    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-envoy\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960152 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istiod-ca-cert\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960294 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960169    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-credential-socket\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960294 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-data\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960427 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-socket\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960491 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-certs\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960600 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960557    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-credential-socket\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960812 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960788    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-data\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.960850 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.960810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istiod-ca-cert\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.962453 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.962432    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-podinfo\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.962541 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.962481    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-envoy\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.967620 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.967596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-token\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:02.968031 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:02.968012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8925g\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-kube-api-access-8925g\") pod \"e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") " pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:03.035379 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.035358    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:03.168611 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.168567    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"]
Jul 13 21:36:03.173659 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:36:03.173573    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b4b90d_99b3_4e9c_85e5_fc46f8478a5b.slice/crio-f703b689e788fcf520aa17b26cd856eb581110956f0e89e2c46def8b29d6ebfc WatchSource:0}: Error finding container f703b689e788fcf520aa17b26cd856eb581110956f0e89e2c46def8b29d6ebfc: Status 404 returned error can't find the container with id f703b689e788fcf520aa17b26cd856eb581110956f0e89e2c46def8b29d6ebfc
Jul 13 21:36:03.175719 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.175691    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:36:03.175797 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.175754    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:36:03.175797 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.175781    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:36:03.536115 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.536082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" event={"ID":"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b","Type":"ContainerStarted","Data":"3ccdb699d6826a2ab5788a70f69466acb6d612c39200128b30cc38f42cc4cf44"}
Jul 13 21:36:03.536524 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.536137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" event={"ID":"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b","Type":"ContainerStarted","Data":"f703b689e788fcf520aa17b26cd856eb581110956f0e89e2c46def8b29d6ebfc"}
Jul 13 21:36:03.556756 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.556713    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" podStartSLOduration=1.556699054 podStartE2EDuration="1.556699054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:36:03.554143095 +0000 UTC m=+1610.211800100" watchObservedRunningTime="2026-07-13 21:36:03.556699054 +0000 UTC m=+1610.214356060"
Jul 13 21:36:03.987702 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.987665    2564 patch_prober.go:28] interesting pod/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 21:36:03.987856 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.987742    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:36:03.988235 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:03.988203    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:04.036481 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.036454    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:04.041200 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.041182    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:04.058900 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.058875    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"]
Jul 13 21:36:04.063261 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.063237    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.065703 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.065682    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f4738480-serving-cert\""
Jul 13 21:36:04.072495 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.072334    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"]
Jul 13 21:36:04.092751 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.092725    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:04.170766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.170706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8gk\" (UniqueName: \"kubernetes.io/projected/98c26d53-16f4-45aa-9869-42e95cf1a952-kube-api-access-jw8gk\") pod \"maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.170913 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.170792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c26d53-16f4-45aa-9869-42e95cf1a952-openshift-service-ca\") pod \"maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.170913 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.170857    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98c26d53-16f4-45aa-9869-42e95cf1a952-maas-api-tls\") pod \"maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.218918 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.218886    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:04.271781 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.271711    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98c26d53-16f4-45aa-9869-42e95cf1a952-maas-api-tls\") pod \"maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.271888 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.271782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8gk\" (UniqueName: \"kubernetes.io/projected/98c26d53-16f4-45aa-9869-42e95cf1a952-kube-api-access-jw8gk\") pod \"maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.271888 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.271829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c26d53-16f4-45aa-9869-42e95cf1a952-openshift-service-ca\") pod \"maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.272467 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.272444    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c26d53-16f4-45aa-9869-42e95cf1a952-openshift-service-ca\") pod \"maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.274467 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.274445    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98c26d53-16f4-45aa-9869-42e95cf1a952-maas-api-tls\") pod \"maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.280725 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.280704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8gk\" (UniqueName: \"kubernetes.io/projected/98c26d53-16f4-45aa-9869-42e95cf1a952-kube-api-access-jw8gk\") pod \"maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.377428 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.377406    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:04.507092 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.507069    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"]
Jul 13 21:36:04.508703 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:36:04.508664    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c26d53_16f4_45aa_9869_42e95cf1a952.slice/crio-711fe3d164c7f1ff24c86f1b0713d7f0a7390e3f5d7cfccb232020b0c451fb22 WatchSource:0}: Error finding container 711fe3d164c7f1ff24c86f1b0713d7f0a7390e3f5d7cfccb232020b0c451fb22: Status 404 returned error can't find the container with id 711fe3d164c7f1ff24c86f1b0713d7f0a7390e3f5d7cfccb232020b0c451fb22
Jul 13 21:36:04.510115 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.510099    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:36:04.543204 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.543134    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl" event={"ID":"98c26d53-16f4-45aa-9869-42e95cf1a952","Type":"ContainerStarted","Data":"711fe3d164c7f1ff24c86f1b0713d7f0a7390e3f5d7cfccb232020b0c451fb22"}
Jul 13 21:36:04.543510 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.543475    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:04.544388 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.544356    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:04.573676 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.573562    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:04.635202 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:04.635172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:05.550684 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:05.550642    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl" event={"ID":"98c26d53-16f4-45aa-9869-42e95cf1a952","Type":"ContainerStarted","Data":"da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363"}
Jul 13 21:36:05.551119 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:05.550710    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:05.566810 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:05.566766    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl" podStartSLOduration=1.139038793 podStartE2EDuration="1.566750745s" totalImagesPullingTime="427.711952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:36:05.5653118 +0000 UTC m=+1612.222968808" watchObservedRunningTime="2026-07-13 21:36:05.566750745 +0000 UTC m=+1612.224407750"
Jul 13 21:36:07.421838 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.421811    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:36:07.499401 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499370    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-certs\") pod \"cc0bdc11-75f8-4bda-889b-0339d0248f92\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") "
Jul 13 21:36:07.499542 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499419    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-podinfo\") pod \"cc0bdc11-75f8-4bda-889b-0339d0248f92\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") "
Jul 13 21:36:07.499542 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499461    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-envoy\") pod \"cc0bdc11-75f8-4bda-889b-0339d0248f92\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") "
Jul 13 21:36:07.499542 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499511    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-token\") pod \"cc0bdc11-75f8-4bda-889b-0339d0248f92\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") "
Jul 13 21:36:07.499707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499567    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0bdc11-75f8-4bda-889b-0339d0248f92-istiod-ca-cert\") pod \"cc0bdc11-75f8-4bda-889b-0339d0248f92\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") "
Jul 13 21:36:07.499707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499635    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-data\") pod \"cc0bdc11-75f8-4bda-889b-0339d0248f92\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") "
Jul 13 21:36:07.499707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499657    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc0bdc11-75f8-4bda-889b-0339d0248f92" (UID: "cc0bdc11-75f8-4bda-889b-0339d0248f92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:36:07.499707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499668    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-credential-socket\") pod \"cc0bdc11-75f8-4bda-889b-0339d0248f92\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") "
Jul 13 21:36:07.499707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499698    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-socket\") pod \"cc0bdc11-75f8-4bda-889b-0339d0248f92\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") "
Jul 13 21:36:07.499964 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499734    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl72g\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-kube-api-access-jl72g\") pod \"cc0bdc11-75f8-4bda-889b-0339d0248f92\" (UID: \"cc0bdc11-75f8-4bda-889b-0339d0248f92\") "
Jul 13 21:36:07.499964 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499939    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc0bdc11-75f8-4bda-889b-0339d0248f92" (UID: "cc0bdc11-75f8-4bda-889b-0339d0248f92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:36:07.500067 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499986    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:07.500126 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.499997    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0bdc11-75f8-4bda-889b-0339d0248f92-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc0bdc11-75f8-4bda-889b-0339d0248f92" (UID: "cc0bdc11-75f8-4bda-889b-0339d0248f92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:36:07.500126 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.500068    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc0bdc11-75f8-4bda-889b-0339d0248f92" (UID: "cc0bdc11-75f8-4bda-889b-0339d0248f92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:36:07.501360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.501327    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc0bdc11-75f8-4bda-889b-0339d0248f92" (UID: "cc0bdc11-75f8-4bda-889b-0339d0248f92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:36:07.501984 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.501960    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc0bdc11-75f8-4bda-889b-0339d0248f92" (UID: "cc0bdc11-75f8-4bda-889b-0339d0248f92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:36:07.502288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.502271    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-kube-api-access-jl72g" (OuterVolumeSpecName: "kube-api-access-jl72g") pod "cc0bdc11-75f8-4bda-889b-0339d0248f92" (UID: "cc0bdc11-75f8-4bda-889b-0339d0248f92"). InnerVolumeSpecName "kube-api-access-jl72g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:36:07.502413 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.502391    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc0bdc11-75f8-4bda-889b-0339d0248f92" (UID: "cc0bdc11-75f8-4bda-889b-0339d0248f92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:36:07.502468 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.502423    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc0bdc11-75f8-4bda-889b-0339d0248f92" (UID: "cc0bdc11-75f8-4bda-889b-0339d0248f92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:36:07.561013 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.560981    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerID="698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c" exitCode=0
Jul 13 21:36:07.561128 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.561038    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"
Jul 13 21:36:07.561128 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.561050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" event={"ID":"cc0bdc11-75f8-4bda-889b-0339d0248f92","Type":"ContainerDied","Data":"698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c"}
Jul 13 21:36:07.561248 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.561137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z" event={"ID":"cc0bdc11-75f8-4bda-889b-0339d0248f92","Type":"ContainerDied","Data":"6e15ab4e39b95cd7d0ac0fb4b69fca8939a246eca729a0d9ab5c4f4f3a47069f"}
Jul 13 21:36:07.561248 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.561172    2564 scope.go:117] "RemoveContainer" containerID="698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c"
Jul 13 21:36:07.569994 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.569978    2564 scope.go:117] "RemoveContainer" containerID="0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94"
Jul 13 21:36:07.577942 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.577921    2564 scope.go:117] "RemoveContainer" containerID="698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c"
Jul 13 21:36:07.578155 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:36:07.578139    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c\": container with ID starting with 698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c not found: ID does not exist" containerID="698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c"
Jul 13 21:36:07.578203 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.578160    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c"} err="failed to get container status \"698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c\": rpc error: code = NotFound desc = could not find container \"698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c\": container with ID starting with 698456aca22e3dec7aa3ba600ee73350b3f2e3a4e7b9f7ab4bbb7bd43db8784c not found: ID does not exist"
Jul 13 21:36:07.578203 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.578175    2564 scope.go:117] "RemoveContainer" containerID="0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94"
Jul 13 21:36:07.578533 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:36:07.578483    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94\": container with ID starting with 0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94 not found: ID does not exist" containerID="0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94"
Jul 13 21:36:07.578533 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.578517    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94"} err="failed to get container status \"0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94\": rpc error: code = NotFound desc = could not find container \"0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94\": container with ID starting with 0fd4033bdb8b73a1fbf2c7e535b7943e774b129f71d1f0d563e0e06571a07a94 not found: ID does not exist"
Jul 13 21:36:07.580739 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.580719    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"]
Jul 13 21:36:07.584147 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.584127    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6f1f77b0-openshift-default-5688ccbdf-mgj7z"]
Jul 13 21:36:07.601199 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.601176    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0bdc11-75f8-4bda-889b-0339d0248f92-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:07.601199 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.601195    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:07.601307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.601203    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:07.601307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.601214    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:07.601307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.601223    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl72g\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-kube-api-access-jl72g\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:07.601307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.601231    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:07.601307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.601238    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:07.601307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.601246    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0bdc11-75f8-4bda-889b-0339d0248f92-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:07.822736 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:07.822712    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" path="/var/lib/kubelet/pods/cc0bdc11-75f8-4bda-889b-0339d0248f92/volumes"
Jul 13 21:36:08.523185 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.523152    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"]
Jul 13 21:36:08.523537 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.523507    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy"
Jul 13 21:36:08.523537 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.523533    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy"
Jul 13 21:36:08.523671 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.523657    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy"
Jul 13 21:36:08.523671 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.523672    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy"
Jul 13 21:36:08.526634 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.526619    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"
Jul 13 21:36:08.528924 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.528902    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:36:08.534415 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.533847    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"]
Jul 13 21:36:08.609117 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.609089    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5713c170-6f2a-4cd6-8839-dff8df377ba8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd\" (UID: \"5713c170-6f2a-4cd6-8839-dff8df377ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"
Jul 13 21:36:08.709707 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.709675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5713c170-6f2a-4cd6-8839-dff8df377ba8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd\" (UID: \"5713c170-6f2a-4cd6-8839-dff8df377ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"
Jul 13 21:36:08.710362 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.710341    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5713c170-6f2a-4cd6-8839-dff8df377ba8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd\" (UID: \"5713c170-6f2a-4cd6-8839-dff8df377ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"
Jul 13 21:36:08.737737 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.737701    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:08.788947 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.788868    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:08.839072 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.839028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"
Jul 13 21:36:08.972780 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:08.972753    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"]
Jul 13 21:36:08.974270 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:36:08.974243    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5713c170_6f2a_4cd6_8839_dff8df377ba8.slice/crio-29945b246852ce947aec01fd44378765a5bdf37cedc3cffb64ee618fd76bf13b WatchSource:0}: Error finding container 29945b246852ce947aec01fd44378765a5bdf37cedc3cffb64ee618fd76bf13b: Status 404 returned error can't find the container with id 29945b246852ce947aec01fd44378765a5bdf37cedc3cffb64ee618fd76bf13b
Jul 13 21:36:09.572739 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:09.572708    2564 generic.go:356] "Generic (PLEG): container finished" podID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerID="9f6d236def06a3250badc78d830430264fdea78e3e6bcc0307a8baf4b50467e9" exitCode=7
Jul 13 21:36:09.573086 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:09.572791    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd" event={"ID":"5713c170-6f2a-4cd6-8839-dff8df377ba8","Type":"ContainerDied","Data":"9f6d236def06a3250badc78d830430264fdea78e3e6bcc0307a8baf4b50467e9"}
Jul 13 21:36:09.573086 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:09.572844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd" event={"ID":"5713c170-6f2a-4cd6-8839-dff8df377ba8","Type":"ContainerStarted","Data":"29945b246852ce947aec01fd44378765a5bdf37cedc3cffb64ee618fd76bf13b"}
Jul 13 21:36:09.573086 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:09.573034    2564 scope.go:117] "RemoveContainer" containerID="9f6d236def06a3250badc78d830430264fdea78e3e6bcc0307a8baf4b50467e9"
Jul 13 21:36:10.579385 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:10.579347    2564 generic.go:356] "Generic (PLEG): container finished" podID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerID="ff902b3ce50fe69e33faf1b18655627e8eb5377459c186280577dd2ee626038a" exitCode=7
Jul 13 21:36:10.579806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:10.579429    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd" event={"ID":"5713c170-6f2a-4cd6-8839-dff8df377ba8","Type":"ContainerDied","Data":"ff902b3ce50fe69e33faf1b18655627e8eb5377459c186280577dd2ee626038a"}
Jul 13 21:36:10.579806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:10.579483    2564 scope.go:117] "RemoveContainer" containerID="9f6d236def06a3250badc78d830430264fdea78e3e6bcc0307a8baf4b50467e9"
Jul 13 21:36:10.579806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:10.579635    2564 scope.go:117] "RemoveContainer" containerID="ff902b3ce50fe69e33faf1b18655627e8eb5377459c186280577dd2ee626038a"
Jul 13 21:36:10.579911 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:36:10.579818    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd_odh-ai-gateway-infra(5713c170-6f2a-4cd6-8839-dff8df377ba8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8"
Jul 13 21:36:11.560144 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:11.560117    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:36:11.585836 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:11.585804    2564 scope.go:117] "RemoveContainer" containerID="ff902b3ce50fe69e33faf1b18655627e8eb5377459c186280577dd2ee626038a"
Jul 13 21:36:11.586219 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:36:11.586010    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd_odh-ai-gateway-infra(5713c170-6f2a-4cd6-8839-dff8df377ba8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8"
Jul 13 21:36:12.235429 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:12.235390    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:19.545364 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:19.545326    2564 patch_prober.go:28] interesting pod/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 13 21:36:19.545805 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:19.545398    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 13 21:36:19.624199 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:19.624171    2564 generic.go:356] "Generic (PLEG): container finished" podID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerID="3ccdb699d6826a2ab5788a70f69466acb6d612c39200128b30cc38f42cc4cf44" exitCode=0
Jul 13 21:36:19.624329 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:19.624269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" event={"ID":"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b","Type":"ContainerDied","Data":"3ccdb699d6826a2ab5788a70f69466acb6d612c39200128b30cc38f42cc4cf44"}
Jul 13 21:36:19.624669 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:19.624653    2564 scope.go:117] "RemoveContainer" containerID="3ccdb699d6826a2ab5788a70f69466acb6d612c39200128b30cc38f42cc4cf44"
Jul 13 21:36:19.625757 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:19.625722    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:36:19.625836 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:19.625817    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:36:19.625879 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:19.625854    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:36:20.630310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:20.630279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" event={"ID":"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b","Type":"ContainerStarted","Data":"ccf45c3884aaffe09709077287c20bce0e3adc97891d347c04e559c5fb17a879"}
Jul 13 21:36:21.036219 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:21.036185    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:21.036363 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:21.036341    2564 patch_prober.go:28] interesting pod/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 13 21:36:21.036401 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:21.036386    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 13 21:36:22.036772 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:22.036741    2564 patch_prober.go:28] interesting pod/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 13 21:36:22.037123 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:22.036792    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 13 21:36:23.035974 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:23.035943    2564 patch_prober.go:28] interesting pod/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 13 21:36:23.036148 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:23.035996    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 13 21:36:24.036579 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:24.036541    2564 patch_prober.go:28] interesting pod/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 13 21:36:24.036977 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:24.036615    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 13 21:36:25.040116 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:25.040082    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:25.040568 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:25.040471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:25.041021 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:25.041003    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:36:25.816024 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:25.815993    2564 scope.go:117] "RemoveContainer" containerID="ff902b3ce50fe69e33faf1b18655627e8eb5377459c186280577dd2ee626038a"
Jul 13 21:36:26.661155 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:26.661126    2564 generic.go:356] "Generic (PLEG): container finished" podID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerID="41101aa8b2ab365c37eb56f9a38362a290d8921427ca7aacd88b1961f28c380f" exitCode=0
Jul 13 21:36:26.661555 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:26.661195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd" event={"ID":"5713c170-6f2a-4cd6-8839-dff8df377ba8","Type":"ContainerDied","Data":"41101aa8b2ab365c37eb56f9a38362a290d8921427ca7aacd88b1961f28c380f"}
Jul 13 21:36:26.661555 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:26.661254    2564 scope.go:117] "RemoveContainer" containerID="ff902b3ce50fe69e33faf1b18655627e8eb5377459c186280577dd2ee626038a"
Jul 13 21:36:27.687550 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:27.687518    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"]
Jul 13 21:36:27.804911 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:27.804891    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"
Jul 13 21:36:27.973773 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:27.973700    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5713c170-6f2a-4cd6-8839-dff8df377ba8-maas-api-service-ca\") pod \"5713c170-6f2a-4cd6-8839-dff8df377ba8\" (UID: \"5713c170-6f2a-4cd6-8839-dff8df377ba8\") "
Jul 13 21:36:27.974106 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:27.974082    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5713c170-6f2a-4cd6-8839-dff8df377ba8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5713c170-6f2a-4cd6-8839-dff8df377ba8" (UID: "5713c170-6f2a-4cd6-8839-dff8df377ba8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:36:28.074398 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:28.074374    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5713c170-6f2a-4cd6-8839-dff8df377ba8-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:36:28.673752 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:28.673722    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"
Jul 13 21:36:28.673752 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:28.673731    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd" event={"ID":"5713c170-6f2a-4cd6-8839-dff8df377ba8","Type":"ContainerDied","Data":"29945b246852ce947aec01fd44378765a5bdf37cedc3cffb64ee618fd76bf13b"}
Jul 13 21:36:28.673953 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:28.673789    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29945b246852ce947aec01fd44378765a5bdf37cedc3cffb64ee618fd76bf13b"
Jul 13 21:36:28.702367 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:28.702342    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"]
Jul 13 21:36:28.706361 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:28.706339    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-fk9dd"]
Jul 13 21:36:29.821260 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:36:29.821223    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" path="/var/lib/kubelet/pods/5713c170-6f2a-4cd6-8839-dff8df377ba8/volumes"
Jul 13 21:37:08.567665 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.567626    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"]
Jul 13 21:37:08.568109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.567924    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy"
Jul 13 21:37:08.568109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.567935    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0bdc11-75f8-4bda-889b-0339d0248f92" containerName="istio-proxy"
Jul 13 21:37:08.568109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.567944    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerName="revoke-keys"
Jul 13 21:37:08.568109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.567949    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerName="revoke-keys"
Jul 13 21:37:08.568109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.567968    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerName="revoke-keys"
Jul 13 21:37:08.568109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.567973    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerName="revoke-keys"
Jul 13 21:37:08.568109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.568020    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerName="revoke-keys"
Jul 13 21:37:08.568109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.568026    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerName="revoke-keys"
Jul 13 21:37:08.570996 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.570978    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"
Jul 13 21:37:08.573319 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.573294    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:37:08.578811 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.577946    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"]
Jul 13 21:37:08.639538 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.639505    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"]
Jul 13 21:37:08.639873 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.639844    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy" containerID="cri-o://ccf45c3884aaffe09709077287c20bce0e3adc97891d347c04e559c5fb17a879" gracePeriod=30
Jul 13 21:37:08.664095 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.664065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169672e1-1952-4ec7-9b48-aa62bd679ab6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4738480-smsmh\" (UID: \"169672e1-1952-4ec7-9b48-aa62bd679ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"
Jul 13 21:37:08.665982 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.665960    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:08.765373 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.765339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169672e1-1952-4ec7-9b48-aa62bd679ab6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4738480-smsmh\" (UID: \"169672e1-1952-4ec7-9b48-aa62bd679ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"
Jul 13 21:37:08.766002 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.765978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169672e1-1952-4ec7-9b48-aa62bd679ab6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4738480-smsmh\" (UID: \"169672e1-1952-4ec7-9b48-aa62bd679ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"
Jul 13 21:37:08.768442 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.768401    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:08.883708 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:08.883644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"
Jul 13 21:37:09.011310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:09.011253    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"]
Jul 13 21:37:09.013711 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:37:09.013684    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169672e1_1952_4ec7_9b48_aa62bd679ab6.slice/crio-437e03a77cece9e497d3fa2f5dd5f26a12283c29a77ca33d3f3ec0cae57160bc WatchSource:0}: Error finding container 437e03a77cece9e497d3fa2f5dd5f26a12283c29a77ca33d3f3ec0cae57160bc: Status 404 returned error can't find the container with id 437e03a77cece9e497d3fa2f5dd5f26a12283c29a77ca33d3f3ec0cae57160bc
Jul 13 21:37:09.856334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:09.856307    2564 generic.go:356] "Generic (PLEG): container finished" podID="169672e1-1952-4ec7-9b48-aa62bd679ab6" containerID="8f9e92498601e1ddfc12813ebdb0bd5e134b70866056863ac7432a89f145da13" exitCode=0
Jul 13 21:37:09.856658 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:09.856393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh" event={"ID":"169672e1-1952-4ec7-9b48-aa62bd679ab6","Type":"ContainerDied","Data":"8f9e92498601e1ddfc12813ebdb0bd5e134b70866056863ac7432a89f145da13"}
Jul 13 21:37:09.856658 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:09.856438    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh" event={"ID":"169672e1-1952-4ec7-9b48-aa62bd679ab6","Type":"ContainerStarted","Data":"437e03a77cece9e497d3fa2f5dd5f26a12283c29a77ca33d3f3ec0cae57160bc"}
Jul 13 21:37:10.042319 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:10.042290    2564 patch_prober.go:28] interesting pod/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 21:37:10.042456 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:10.042335    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:37:10.992648 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:10.992627    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"
Jul 13 21:37:11.086131 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:11.086104    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169672e1-1952-4ec7-9b48-aa62bd679ab6-maas-api-service-ca\") pod \"169672e1-1952-4ec7-9b48-aa62bd679ab6\" (UID: \"169672e1-1952-4ec7-9b48-aa62bd679ab6\") "
Jul 13 21:37:11.086457 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:11.086436    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169672e1-1952-4ec7-9b48-aa62bd679ab6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "169672e1-1952-4ec7-9b48-aa62bd679ab6" (UID: "169672e1-1952-4ec7-9b48-aa62bd679ab6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:37:11.187011 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:11.186952    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169672e1-1952-4ec7-9b48-aa62bd679ab6-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:11.866041 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:11.866016    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh"
Jul 13 21:37:11.866041 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:11.866028    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4738480-smsmh" event={"ID":"169672e1-1952-4ec7-9b48-aa62bd679ab6","Type":"ContainerDied","Data":"437e03a77cece9e497d3fa2f5dd5f26a12283c29a77ca33d3f3ec0cae57160bc"}
Jul 13 21:37:11.866202 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:11.866068    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="437e03a77cece9e497d3fa2f5dd5f26a12283c29a77ca33d3f3ec0cae57160bc"
Jul 13 21:37:13.876180 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:13.876155    2564 generic.go:356] "Generic (PLEG): container finished" podID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerID="ccf45c3884aaffe09709077287c20bce0e3adc97891d347c04e559c5fb17a879" exitCode=0
Jul 13 21:37:13.876531 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:13.876224    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" event={"ID":"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b","Type":"ContainerDied","Data":"ccf45c3884aaffe09709077287c20bce0e3adc97891d347c04e559c5fb17a879"}
Jul 13 21:37:13.876531 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:13.876274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l" event={"ID":"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b","Type":"ContainerDied","Data":"f703b689e788fcf520aa17b26cd856eb581110956f0e89e2c46def8b29d6ebfc"}
Jul 13 21:37:13.876531 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:13.876291    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f703b689e788fcf520aa17b26cd856eb581110956f0e89e2c46def8b29d6ebfc"
Jul 13 21:37:13.876531 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:13.876304    2564 scope.go:117] "RemoveContainer" containerID="3ccdb699d6826a2ab5788a70f69466acb6d612c39200128b30cc38f42cc4cf44"
Jul 13 21:37:13.891246 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:13.891225    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:37:14.005979 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.005953    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8925g\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-kube-api-access-8925g\") pod \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") "
Jul 13 21:37:14.006109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006003    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-credential-socket\") pod \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") "
Jul 13 21:37:14.006109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006027    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-data\") pod \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") "
Jul 13 21:37:14.006109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006062    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-podinfo\") pod \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") "
Jul 13 21:37:14.006109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006086    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-certs\") pod \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") "
Jul 13 21:37:14.006509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006117    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-token\") pod \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") "
Jul 13 21:37:14.006509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006140    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istiod-ca-cert\") pod \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") "
Jul 13 21:37:14.006509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006202    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-socket\") pod \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") "
Jul 13 21:37:14.006509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006226    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-envoy\") pod \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\" (UID: \"f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b\") "
Jul 13 21:37:14.006509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006357    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" (UID: "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:37:14.006509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006409    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" (UID: "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:37:14.006509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006490    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.006509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006507    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.006908 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006765    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" (UID: "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:37:14.006908 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006784    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" (UID: "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:37:14.007017 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.006970    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" (UID: "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:37:14.008365 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.008341    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" (UID: "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:37:14.008490 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.008442    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" (UID: "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:37:14.008793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.008776    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-kube-api-access-8925g" (OuterVolumeSpecName: "kube-api-access-8925g") pod "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" (UID: "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b"). InnerVolumeSpecName "kube-api-access-8925g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:37:14.008857 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.008780    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" (UID: "f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:37:14.107010 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.106988    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8925g\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-kube-api-access-8925g\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.107010 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.107010    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.107156 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.107019    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.107156 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.107026    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.107156 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.107034    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.107156 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.107042    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.107156 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.107049    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.157626 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.157570    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"]
Jul 13 21:37:14.158047 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.157951    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl" podUID="98c26d53-16f4-45aa-9869-42e95cf1a952" containerName="maas-api" containerID="cri-o://da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363" gracePeriod=30
Jul 13 21:37:14.183212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.183175    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:14.389541 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.389517    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:37:14.510122 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.510094    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw8gk\" (UniqueName: \"kubernetes.io/projected/98c26d53-16f4-45aa-9869-42e95cf1a952-kube-api-access-jw8gk\") pod \"98c26d53-16f4-45aa-9869-42e95cf1a952\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") "
Jul 13 21:37:14.510255 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.510147    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c26d53-16f4-45aa-9869-42e95cf1a952-openshift-service-ca\") pod \"98c26d53-16f4-45aa-9869-42e95cf1a952\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") "
Jul 13 21:37:14.510255 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.510183    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98c26d53-16f4-45aa-9869-42e95cf1a952-maas-api-tls\") pod \"98c26d53-16f4-45aa-9869-42e95cf1a952\" (UID: \"98c26d53-16f4-45aa-9869-42e95cf1a952\") "
Jul 13 21:37:14.510562 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.510531    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c26d53-16f4-45aa-9869-42e95cf1a952-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98c26d53-16f4-45aa-9869-42e95cf1a952" (UID: "98c26d53-16f4-45aa-9869-42e95cf1a952"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:37:14.512195 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.512172    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c26d53-16f4-45aa-9869-42e95cf1a952-kube-api-access-jw8gk" (OuterVolumeSpecName: "kube-api-access-jw8gk") pod "98c26d53-16f4-45aa-9869-42e95cf1a952" (UID: "98c26d53-16f4-45aa-9869-42e95cf1a952"). InnerVolumeSpecName "kube-api-access-jw8gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:37:14.512277 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.512235    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c26d53-16f4-45aa-9869-42e95cf1a952-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98c26d53-16f4-45aa-9869-42e95cf1a952" (UID: "98c26d53-16f4-45aa-9869-42e95cf1a952"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:37:14.589110 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.589051    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:14.611096 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.611075    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw8gk\" (UniqueName: \"kubernetes.io/projected/98c26d53-16f4-45aa-9869-42e95cf1a952-kube-api-access-jw8gk\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.611096 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.611095    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c26d53-16f4-45aa-9869-42e95cf1a952-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.611220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.611105    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98c26d53-16f4-45aa-9869-42e95cf1a952-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:14.882376 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.882323    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"
Jul 13 21:37:14.883748 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.883721    2564 generic.go:356] "Generic (PLEG): container finished" podID="98c26d53-16f4-45aa-9869-42e95cf1a952" containerID="da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363" exitCode=0
Jul 13 21:37:14.883851 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.883769    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"
Jul 13 21:37:14.883851 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.883786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl" event={"ID":"98c26d53-16f4-45aa-9869-42e95cf1a952","Type":"ContainerDied","Data":"da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363"}
Jul 13 21:37:14.883851 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.883836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl" event={"ID":"98c26d53-16f4-45aa-9869-42e95cf1a952","Type":"ContainerDied","Data":"711fe3d164c7f1ff24c86f1b0713d7f0a7390e3f5d7cfccb232020b0c451fb22"}
Jul 13 21:37:14.883978 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.883867    2564 scope.go:117] "RemoveContainer" containerID="da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363"
Jul 13 21:37:14.892483 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.892465    2564 scope.go:117] "RemoveContainer" containerID="da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363"
Jul 13 21:37:14.892793 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:37:14.892772    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363\": container with ID starting with da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363 not found: ID does not exist" containerID="da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363"
Jul 13 21:37:14.892891 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.892802    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363"} err="failed to get container status \"da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363\": rpc error: code = NotFound desc = could not find container \"da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363\": container with ID starting with da034cf635949216e0d2d596cd98201a0b07d03c0fcd16cc12cb9387230ca363 not found: ID does not exist"
Jul 13 21:37:14.914894 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.914857    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"]
Jul 13 21:37:14.916538 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.916518    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f4738480-gw-openshift-default-75d8f78f95-9st7l"]
Jul 13 21:37:14.926181 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.926142    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"]
Jul 13 21:37:14.927786 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:14.927767    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4738480-7c4f8595bb-8s6tl"]
Jul 13 21:37:15.390401 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:15.390343    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:15.821154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:15.821126    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98c26d53-16f4-45aa-9869-42e95cf1a952" path="/var/lib/kubelet/pods/98c26d53-16f4-45aa-9869-42e95cf1a952/volumes"
Jul 13 21:37:15.821509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:15.821496    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" path="/var/lib/kubelet/pods/f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b/volumes"
Jul 13 21:37:20.361195 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.361144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:20.463792 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.463761    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:20.527552 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527527    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"]
Jul 13 21:37:20.527896 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527880    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c26d53-16f4-45aa-9869-42e95cf1a952" containerName="maas-api"
Jul 13 21:37:20.527980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527899    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c26d53-16f4-45aa-9869-42e95cf1a952" containerName="maas-api"
Jul 13 21:37:20.527980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527922    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy"
Jul 13 21:37:20.527980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527930    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy"
Jul 13 21:37:20.527980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527942    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerName="revoke-keys"
Jul 13 21:37:20.527980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527951    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerName="revoke-keys"
Jul 13 21:37:20.527980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527968    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy"
Jul 13 21:37:20.527980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527976    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy"
Jul 13 21:37:20.528308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527991    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169672e1-1952-4ec7-9b48-aa62bd679ab6" containerName="revoke-keys"
Jul 13 21:37:20.528308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.527999    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="169672e1-1952-4ec7-9b48-aa62bd679ab6" containerName="revoke-keys"
Jul 13 21:37:20.528308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.528091    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy"
Jul 13 21:37:20.528308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.528105    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5b4b90d-99b3-4e9c-85e5-fc46f8478a5b" containerName="istio-proxy"
Jul 13 21:37:20.528308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.528116    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5713c170-6f2a-4cd6-8839-dff8df377ba8" containerName="revoke-keys"
Jul 13 21:37:20.528308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.528126    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="169672e1-1952-4ec7-9b48-aa62bd679ab6" containerName="revoke-keys"
Jul 13 21:37:20.528308 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.528134    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="98c26d53-16f4-45aa-9869-42e95cf1a952" containerName="maas-api"
Jul 13 21:37:20.532458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.532438    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.534909 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.534886    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-057af81a-openshift-default-dockercfg-h6785\""
Jul 13 21:37:20.542730 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.542708    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"]
Jul 13 21:37:20.563656 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.563615    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:20.654822 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.654740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-certs\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.654822 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.654780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-envoy\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.654822 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.654803    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istiod-ca-cert\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.655037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.654907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxgww\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-kube-api-access-nxgww\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.655037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.654940    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-token\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.655037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.654961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-socket\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.655037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.654981    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-data\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.655037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.654995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-podinfo\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.655037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.655024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-credential-socket\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.663469 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.663441    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:20.756017 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.755994    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-certs\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-envoy\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istiod-ca-cert\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxgww\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-kube-api-access-nxgww\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-token\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756297 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-socket\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756297 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-data\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756297 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-podinfo\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756297 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756226    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-credential-socket\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756500 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-certs\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756562 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-socket\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756653 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756621    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-data\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.756695 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.756653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-credential-socket\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.757132 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.757023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istiod-ca-cert\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.758394 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.758372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-envoy\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.758748 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.758730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-podinfo\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.763833 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.763808    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-token\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.763919 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.763840    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxgww\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-kube-api-access-nxgww\") pod \"e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") " pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.843520 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.843496    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:20.979868 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.979847    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"]
Jul 13 21:37:20.981809 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:37:20.981783    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c265e6_647d_4c8e_ad6c_92477f249e5c.slice/crio-6a16a27490d0d07415dd1e997d0873749ee06bd7e9ae2c31af9a0e2d5d0d1c3f WatchSource:0}: Error finding container 6a16a27490d0d07415dd1e997d0873749ee06bd7e9ae2c31af9a0e2d5d0d1c3f: Status 404 returned error can't find the container with id 6a16a27490d0d07415dd1e997d0873749ee06bd7e9ae2c31af9a0e2d5d0d1c3f
Jul 13 21:37:20.984050 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.984013    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:37:20.984140 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.984096    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:37:20.984188 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:20.984144    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:37:21.664749 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.664713    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:21.765628 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.765559    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:21.796179 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.796154    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"]
Jul 13 21:37:21.799357 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.799340    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:21.801745 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.801728    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-057af81a-serving-cert\""
Jul 13 21:37:21.809807 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.809788    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"]
Jul 13 21:37:21.865338 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.865280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede2666-9aaf-481e-941c-529c3ad2689a-maas-api-tls\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:21.865513 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.865450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede2666-9aaf-481e-941c-529c3ad2689a-openshift-service-ca\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:21.865613 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.865547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft8td\" (UniqueName: \"kubernetes.io/projected/cede2666-9aaf-481e-941c-529c3ad2689a-kube-api-access-ft8td\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:21.896300 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.896267    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:21.918842 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.918773    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" event={"ID":"d2c265e6-647d-4c8e-ad6c-92477f249e5c","Type":"ContainerStarted","Data":"8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b"}
Jul 13 21:37:21.918842 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.918818    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" event={"ID":"d2c265e6-647d-4c8e-ad6c-92477f249e5c","Type":"ContainerStarted","Data":"6a16a27490d0d07415dd1e997d0873749ee06bd7e9ae2c31af9a0e2d5d0d1c3f"}
Jul 13 21:37:21.938986 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.938946    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" podStartSLOduration=1.9389310050000002 podStartE2EDuration="1.938931005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:37:21.937134519 +0000 UTC m=+1688.594791524" watchObservedRunningTime="2026-07-13 21:37:21.938931005 +0000 UTC m=+1688.596588011"
Jul 13 21:37:21.966111 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.966083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede2666-9aaf-481e-941c-529c3ad2689a-maas-api-tls\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:21.966206 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.966139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede2666-9aaf-481e-941c-529c3ad2689a-openshift-service-ca\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:21.966206 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.966178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8td\" (UniqueName: \"kubernetes.io/projected/cede2666-9aaf-481e-941c-529c3ad2689a-kube-api-access-ft8td\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:21.966286 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:37:21.966229    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-serving-cert: secret "maas-api-e2e-ait-057af81a-serving-cert" not found
Jul 13 21:37:21.966336 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:37:21.966324    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cede2666-9aaf-481e-941c-529c3ad2689a-maas-api-tls podName:cede2666-9aaf-481e-941c-529c3ad2689a nodeName:}" failed. No retries permitted until 2026-07-13 21:37:22.466306627 +0000 UTC m=+1689.123963611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cede2666-9aaf-481e-941c-529c3ad2689a-maas-api-tls") pod "maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l" (UID: "cede2666-9aaf-481e-941c-529c3ad2689a") : secret "maas-api-e2e-ait-057af81a-serving-cert" not found
Jul 13 21:37:21.966861 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.966845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede2666-9aaf-481e-941c-529c3ad2689a-openshift-service-ca\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:21.974973 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:21.974947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft8td\" (UniqueName: \"kubernetes.io/projected/cede2666-9aaf-481e-941c-529c3ad2689a-kube-api-access-ft8td\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:22.453789 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.453755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:22.469750 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.469720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede2666-9aaf-481e-941c-529c3ad2689a-maas-api-tls\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:22.472133 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.472114    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede2666-9aaf-481e-941c-529c3ad2689a-maas-api-tls\") pod \"maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:22.548856 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.548828    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:22.710602 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.710505    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:22.843921 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.843885    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:22.848748 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.848723    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:22.853766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.853734    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"]
Jul 13 21:37:22.856233 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:37:22.856206    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcede2666_9aaf_481e_941c_529c3ad2689a.slice/crio-85d7db2424ed64a69f1a9a7f26256ec46ef4f081b0c9b46b35010ee8eb913821 WatchSource:0}: Error finding container 85d7db2424ed64a69f1a9a7f26256ec46ef4f081b0c9b46b35010ee8eb913821: Status 404 returned error can't find the container with id 85d7db2424ed64a69f1a9a7f26256ec46ef4f081b0c9b46b35010ee8eb913821
Jul 13 21:37:22.924572 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.924546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l" event={"ID":"cede2666-9aaf-481e-941c-529c3ad2689a","Type":"ContainerStarted","Data":"85d7db2424ed64a69f1a9a7f26256ec46ef4f081b0c9b46b35010ee8eb913821"}
Jul 13 21:37:22.924777 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.924759    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:22.925504 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:22.925489    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:23.931526 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:23.931437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l" event={"ID":"cede2666-9aaf-481e-941c-529c3ad2689a","Type":"ContainerStarted","Data":"b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054"}
Jul 13 21:37:23.931896 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:23.931576    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:23.947858 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:23.947819    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l" podStartSLOduration=2.135280445 podStartE2EDuration="2.947805928s" totalImagesPullingTime="812.525483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:37:23.94674713 +0000 UTC m=+1690.604404137" watchObservedRunningTime="2026-07-13 21:37:23.947805928 +0000 UTC m=+1690.605462933"
Jul 13 21:37:24.078116 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:24.078084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:26.735098 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:26.735052    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"]
Jul 13 21:37:26.738554 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:26.738530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"
Jul 13 21:37:26.741170 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:26.741147    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:37:26.745113 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:26.745086    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"]
Jul 13 21:37:26.803359 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:26.803334    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe97a-21cf-4e83-9c1a-948c6b09d3ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-057af81a-5mms9\" (UID: \"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"
Jul 13 21:37:26.904141 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:26.904110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe97a-21cf-4e83-9c1a-948c6b09d3ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-057af81a-5mms9\" (UID: \"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"
Jul 13 21:37:26.904688 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:26.904669    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe97a-21cf-4e83-9c1a-948c6b09d3ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-057af81a-5mms9\" (UID: \"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"
Jul 13 21:37:27.055488 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:27.055455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"
Jul 13 21:37:27.219649 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:27.219628    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"]
Jul 13 21:37:27.222485 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:37:27.222456    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ffe97a_21cf_4e83_9c1a_948c6b09d3ce.slice/crio-a79d39fa382674f3c335139a5836b17a18be15ce7d6975e8cd9c0e88666d0c18 WatchSource:0}: Error finding container a79d39fa382674f3c335139a5836b17a18be15ce7d6975e8cd9c0e88666d0c18: Status 404 returned error can't find the container with id a79d39fa382674f3c335139a5836b17a18be15ce7d6975e8cd9c0e88666d0c18
Jul 13 21:37:27.964453 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:27.964393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9" event={"ID":"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce","Type":"ContainerStarted","Data":"906ed3eeddf8ce60dfa8cca33fdd0a4bbb8efaffc8bb587c64420862c9673eb9"}
Jul 13 21:37:27.964974 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:27.964477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9" event={"ID":"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce","Type":"ContainerStarted","Data":"a79d39fa382674f3c335139a5836b17a18be15ce7d6975e8cd9c0e88666d0c18"}
Jul 13 21:37:27.978526 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:27.978476    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9" podStartSLOduration=1.9784594279999999 podStartE2EDuration="1.978459428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:37:27.978326704 +0000 UTC m=+1694.635983712" watchObservedRunningTime="2026-07-13 21:37:27.978459428 +0000 UTC m=+1694.636116435"
Jul 13 21:37:28.972811 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:28.972774    2564 generic.go:356] "Generic (PLEG): container finished" podID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerID="906ed3eeddf8ce60dfa8cca33fdd0a4bbb8efaffc8bb587c64420862c9673eb9" exitCode=7
Jul 13 21:37:28.972811 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:28.972813    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9" event={"ID":"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce","Type":"ContainerDied","Data":"906ed3eeddf8ce60dfa8cca33fdd0a4bbb8efaffc8bb587c64420862c9673eb9"}
Jul 13 21:37:28.973212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:28.972995    2564 scope.go:117] "RemoveContainer" containerID="906ed3eeddf8ce60dfa8cca33fdd0a4bbb8efaffc8bb587c64420862c9673eb9"
Jul 13 21:37:29.943350 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:29.942741    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:37:29.980233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:29.980197    2564 generic.go:356] "Generic (PLEG): container finished" podID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerID="1cbb637cf8b2c783bc256cca62dce689ab050f7abf95f7eb7cba9d97ce199e3e" exitCode=7
Jul 13 21:37:29.980688 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:29.980268    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9" event={"ID":"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce","Type":"ContainerDied","Data":"1cbb637cf8b2c783bc256cca62dce689ab050f7abf95f7eb7cba9d97ce199e3e"}
Jul 13 21:37:29.980688 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:29.980339    2564 scope.go:117] "RemoveContainer" containerID="906ed3eeddf8ce60dfa8cca33fdd0a4bbb8efaffc8bb587c64420862c9673eb9"
Jul 13 21:37:29.980688 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:29.980528    2564 scope.go:117] "RemoveContainer" containerID="1cbb637cf8b2c783bc256cca62dce689ab050f7abf95f7eb7cba9d97ce199e3e"
Jul 13 21:37:29.980871 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:37:29.980846    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-057af81a-5mms9_odh-ai-gateway-infra(31ffe97a-21cf-4e83-9c1a-948c6b09d3ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce"
Jul 13 21:37:30.986509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:30.986483    2564 scope.go:117] "RemoveContainer" containerID="1cbb637cf8b2c783bc256cca62dce689ab050f7abf95f7eb7cba9d97ce199e3e"
Jul 13 21:37:30.987013 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:37:30.986690    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-057af81a-5mms9_odh-ai-gateway-infra(31ffe97a-21cf-4e83-9c1a-948c6b09d3ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce"
Jul 13 21:37:32.997836 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:32.997808    2564 generic.go:356] "Generic (PLEG): container finished" podID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerID="8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b" exitCode=0
Jul 13 21:37:32.998288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:32.997891    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" event={"ID":"d2c265e6-647d-4c8e-ad6c-92477f249e5c","Type":"ContainerDied","Data":"8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b"}
Jul 13 21:37:32.998288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:32.998240    2564 scope.go:117] "RemoveContainer" containerID="8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b"
Jul 13 21:37:32.999248 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:32.999220    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:37:32.999354 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:32.999305    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:37:32.999416 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:32.999353    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:37:34.004078 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:34.004043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" event={"ID":"d2c265e6-647d-4c8e-ad6c-92477f249e5c","Type":"ContainerStarted","Data":"997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1"}
Jul 13 21:37:34.845376 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:34.845336    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:34.845613 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:34.845535    2564 patch_prober.go:28] interesting pod/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 13 21:37:34.845701 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:34.845605    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 13 21:37:35.844617 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:35.844559    2564 patch_prober.go:28] interesting pod/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 13 21:37:35.845214 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:35.845179    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 13 21:37:36.844536 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:36.844495    2564 patch_prober.go:28] interesting pod/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 13 21:37:36.844768 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:36.844564    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 13 21:37:37.844128 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:37.844084    2564 patch_prober.go:28] interesting pod/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 13 21:37:37.844338 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:37.844151    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 13 21:37:37.924796 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:37.924765    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:38.848676 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:38.848646    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:38.849652 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:38.849635    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:37:44.323071 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:44.323036    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:44.423346 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:44.423306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:44.816509 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:44.816483    2564 scope.go:117] "RemoveContainer" containerID="1cbb637cf8b2c783bc256cca62dce689ab050f7abf95f7eb7cba9d97ce199e3e"
Jul 13 21:37:45.521384 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:45.521355    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:45.622160 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:45.622132    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:46.063180 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:46.063148    2564 generic.go:356] "Generic (PLEG): container finished" podID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerID="e9566ffaba70e0d5dc5d79081eb93eb91962b6cd9cd715d5e31c2a094c0b68b8" exitCode=0
Jul 13 21:37:46.063339 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:46.063221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9" event={"ID":"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce","Type":"ContainerDied","Data":"e9566ffaba70e0d5dc5d79081eb93eb91962b6cd9cd715d5e31c2a094c0b68b8"}
Jul 13 21:37:46.063339 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:46.063282    2564 scope.go:117] "RemoveContainer" containerID="1cbb637cf8b2c783bc256cca62dce689ab050f7abf95f7eb7cba9d97ce199e3e"
Jul 13 21:37:47.086737 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:47.086700    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"]
Jul 13 21:37:47.208538 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:47.208517    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"
Jul 13 21:37:47.368126 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:47.368061    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe97a-21cf-4e83-9c1a-948c6b09d3ce-maas-api-service-ca\") pod \"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce\" (UID: \"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce\") "
Jul 13 21:37:47.368392 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:47.368371    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ffe97a-21cf-4e83-9c1a-948c6b09d3ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" (UID: "31ffe97a-21cf-4e83-9c1a-948c6b09d3ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:37:47.469332 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:47.469297    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ffe97a-21cf-4e83-9c1a-948c6b09d3ce-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:37:48.076876 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:48.076842    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"
Jul 13 21:37:48.076876 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:48.076861    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9" event={"ID":"31ffe97a-21cf-4e83-9c1a-948c6b09d3ce","Type":"ContainerDied","Data":"a79d39fa382674f3c335139a5836b17a18be15ce7d6975e8cd9c0e88666d0c18"}
Jul 13 21:37:48.077138 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:48.076903    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a79d39fa382674f3c335139a5836b17a18be15ce7d6975e8cd9c0e88666d0c18"
Jul 13 21:37:48.104296 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:48.104229    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"]
Jul 13 21:37:48.107737 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:48.107716    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-5mms9"]
Jul 13 21:37:49.821258 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:49.821214    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" path="/var/lib/kubelet/pods/31ffe97a-21cf-4e83-9c1a-948c6b09d3ce/volumes"
Jul 13 21:37:57.299196 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:57.299160    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:57.400757 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:37:57.400729    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:26.778513 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.778470    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"]
Jul 13 21:38:26.779099 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.778807    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerName="revoke-keys"
Jul 13 21:38:26.779099 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.778819    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerName="revoke-keys"
Jul 13 21:38:26.779099 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.778843    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerName="revoke-keys"
Jul 13 21:38:26.779099 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.778848    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerName="revoke-keys"
Jul 13 21:38:26.779099 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.778895    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerName="revoke-keys"
Jul 13 21:38:26.779099 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.778902    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerName="revoke-keys"
Jul 13 21:38:26.781839 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.781802    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"
Jul 13 21:38:26.784160 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.784138    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:38:26.787780 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.787758    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"]
Jul 13 21:38:26.852340 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.852313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9fd6c9-6901-417e-bcd4-30362aa72eef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-057af81a-99cqh\" (UID: \"1e9fd6c9-6901-417e-bcd4-30362aa72eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"
Jul 13 21:38:26.952644 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.952618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9fd6c9-6901-417e-bcd4-30362aa72eef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-057af81a-99cqh\" (UID: \"1e9fd6c9-6901-417e-bcd4-30362aa72eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"
Jul 13 21:38:26.953199 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:26.953175    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9fd6c9-6901-417e-bcd4-30362aa72eef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-057af81a-99cqh\" (UID: \"1e9fd6c9-6901-417e-bcd4-30362aa72eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"
Jul 13 21:38:27.094233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:27.094155    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"
Jul 13 21:38:27.223405 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:27.223378    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"]
Jul 13 21:38:27.226062 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:38:27.226033    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9fd6c9_6901_417e_bcd4_30362aa72eef.slice/crio-29f5f839d9367324b08941699b4ea9f5da5e858caecf31210d5dd91fc0f556cd WatchSource:0}: Error finding container 29f5f839d9367324b08941699b4ea9f5da5e858caecf31210d5dd91fc0f556cd: Status 404 returned error can't find the container with id 29f5f839d9367324b08941699b4ea9f5da5e858caecf31210d5dd91fc0f556cd
Jul 13 21:38:27.256027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:27.256003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh" event={"ID":"1e9fd6c9-6901-417e-bcd4-30362aa72eef","Type":"ContainerStarted","Data":"29f5f839d9367324b08941699b4ea9f5da5e858caecf31210d5dd91fc0f556cd"}
Jul 13 21:38:28.261705 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:28.261673    2564 generic.go:356] "Generic (PLEG): container finished" podID="1e9fd6c9-6901-417e-bcd4-30362aa72eef" containerID="6378c5b3ddc8368b095252c7dc64ae39faa75ad3b9dabc89a7e05fedbf2847d6" exitCode=0
Jul 13 21:38:28.262023 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:28.261753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh" event={"ID":"1e9fd6c9-6901-417e-bcd4-30362aa72eef","Type":"ContainerDied","Data":"6378c5b3ddc8368b095252c7dc64ae39faa75ad3b9dabc89a7e05fedbf2847d6"}
Jul 13 21:38:29.404172 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:29.404146    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"
Jul 13 21:38:29.469487 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:29.469465    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9fd6c9-6901-417e-bcd4-30362aa72eef-maas-api-service-ca\") pod \"1e9fd6c9-6901-417e-bcd4-30362aa72eef\" (UID: \"1e9fd6c9-6901-417e-bcd4-30362aa72eef\") "
Jul 13 21:38:29.469856 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:29.469835    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9fd6c9-6901-417e-bcd4-30362aa72eef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e9fd6c9-6901-417e-bcd4-30362aa72eef" (UID: "1e9fd6c9-6901-417e-bcd4-30362aa72eef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:38:29.570771 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:29.570713    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9fd6c9-6901-417e-bcd4-30362aa72eef-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:30.271406 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:30.271366    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh" event={"ID":"1e9fd6c9-6901-417e-bcd4-30362aa72eef","Type":"ContainerDied","Data":"29f5f839d9367324b08941699b4ea9f5da5e858caecf31210d5dd91fc0f556cd"}
Jul 13 21:38:30.271572 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:30.271419    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29f5f839d9367324b08941699b4ea9f5da5e858caecf31210d5dd91fc0f556cd"
Jul 13 21:38:30.271572 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:30.271380    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-057af81a-99cqh"
Jul 13 21:38:37.316464 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.316433    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"]
Jul 13 21:38:37.316881 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.316696    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l" podUID="cede2666-9aaf-481e-941c-529c3ad2689a" containerName="maas-api" containerID="cri-o://b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054" gracePeriod=30
Jul 13 21:38:37.346176 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.346149    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:37.555407 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.555379    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:38:37.632243 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.632178    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede2666-9aaf-481e-941c-529c3ad2689a-maas-api-tls\") pod \"cede2666-9aaf-481e-941c-529c3ad2689a\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") "
Jul 13 21:38:37.632243 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.632215    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede2666-9aaf-481e-941c-529c3ad2689a-openshift-service-ca\") pod \"cede2666-9aaf-481e-941c-529c3ad2689a\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") "
Jul 13 21:38:37.632414 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.632251    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft8td\" (UniqueName: \"kubernetes.io/projected/cede2666-9aaf-481e-941c-529c3ad2689a-kube-api-access-ft8td\") pod \"cede2666-9aaf-481e-941c-529c3ad2689a\" (UID: \"cede2666-9aaf-481e-941c-529c3ad2689a\") "
Jul 13 21:38:37.632557 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.632532    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cede2666-9aaf-481e-941c-529c3ad2689a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cede2666-9aaf-481e-941c-529c3ad2689a" (UID: "cede2666-9aaf-481e-941c-529c3ad2689a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:38:37.634369 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.634343    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cede2666-9aaf-481e-941c-529c3ad2689a-kube-api-access-ft8td" (OuterVolumeSpecName: "kube-api-access-ft8td") pod "cede2666-9aaf-481e-941c-529c3ad2689a" (UID: "cede2666-9aaf-481e-941c-529c3ad2689a"). InnerVolumeSpecName "kube-api-access-ft8td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:38:37.634369 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.634353    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cede2666-9aaf-481e-941c-529c3ad2689a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cede2666-9aaf-481e-941c-529c3ad2689a" (UID: "cede2666-9aaf-481e-941c-529c3ad2689a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:38:37.732994 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.732972    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cede2666-9aaf-481e-941c-529c3ad2689a-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:37.732994 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.732991    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cede2666-9aaf-481e-941c-529c3ad2689a-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:37.733121 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:37.733001    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft8td\" (UniqueName: \"kubernetes.io/projected/cede2666-9aaf-481e-941c-529c3ad2689a-kube-api-access-ft8td\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:38.309339 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:38.309299    2564 generic.go:356] "Generic (PLEG): container finished" podID="cede2666-9aaf-481e-941c-529c3ad2689a" containerID="b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054" exitCode=0
Jul 13 21:38:38.309540 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:38.309348    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"
Jul 13 21:38:38.309540 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:38.309387    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l" event={"ID":"cede2666-9aaf-481e-941c-529c3ad2689a","Type":"ContainerDied","Data":"b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054"}
Jul 13 21:38:38.309540 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:38.309456    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l" event={"ID":"cede2666-9aaf-481e-941c-529c3ad2689a","Type":"ContainerDied","Data":"85d7db2424ed64a69f1a9a7f26256ec46ef4f081b0c9b46b35010ee8eb913821"}
Jul 13 21:38:38.309540 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:38.309487    2564 scope.go:117] "RemoveContainer" containerID="b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054"
Jul 13 21:38:38.321723 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:38.321479    2564 scope.go:117] "RemoveContainer" containerID="b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054"
Jul 13 21:38:38.321949 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:38:38.321740    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054\": container with ID starting with b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054 not found: ID does not exist" containerID="b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054"
Jul 13 21:38:38.321949 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:38.321766    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054"} err="failed to get container status \"b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054\": rpc error: code = NotFound desc = could not find container \"b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054\": container with ID starting with b96f3ca3762f31c88c86b62f389ea133b5b2a523a430158f4711017ac619d054 not found: ID does not exist"
Jul 13 21:38:38.326816 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:38.326795    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"]
Jul 13 21:38:38.330388 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:38.330364    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-057af81a-7f7d97ddd8-99s4l"]
Jul 13 21:38:39.822577 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:39.822544    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cede2666-9aaf-481e-941c-529c3ad2689a" path="/var/lib/kubelet/pods/cede2666-9aaf-481e-941c-529c3ad2689a/volumes"
Jul 13 21:38:41.948490 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:41.948454    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:48.171680 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:48.171640    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:48.269864 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:48.269829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:52.908109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:52.908077    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"]
Jul 13 21:38:52.908501 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:52.908278    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy" containerID="cri-o://997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1" gracePeriod=30
Jul 13 21:38:52.930976 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:52.930943    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:53.196079 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196003    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"]
Jul 13 21:38:53.196327 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196315    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cede2666-9aaf-481e-941c-529c3ad2689a" containerName="maas-api"
Jul 13 21:38:53.196371 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196329    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cede2666-9aaf-481e-941c-529c3ad2689a" containerName="maas-api"
Jul 13 21:38:53.196371 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196338    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerName="revoke-keys"
Jul 13 21:38:53.196371 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196343    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerName="revoke-keys"
Jul 13 21:38:53.196371 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196354    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9fd6c9-6901-417e-bcd4-30362aa72eef" containerName="revoke-keys"
Jul 13 21:38:53.196371 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196359    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9fd6c9-6901-417e-bcd4-30362aa72eef" containerName="revoke-keys"
Jul 13 21:38:53.196517 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196408    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ffe97a-21cf-4e83-9c1a-948c6b09d3ce" containerName="revoke-keys"
Jul 13 21:38:53.196517 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196416    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9fd6c9-6901-417e-bcd4-30362aa72eef" containerName="revoke-keys"
Jul 13 21:38:53.196517 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.196424    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cede2666-9aaf-481e-941c-529c3ad2689a" containerName="maas-api"
Jul 13 21:38:53.199282 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.199261    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.201823 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.201801    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ae8a7501-openshift-default-dockercfg-6r69d\""
Jul 13 21:38:53.211215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.211191    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"]
Jul 13 21:38:53.241057 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.241027    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:53.350906 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.350874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istiod-ca-cert\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.351060 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.350912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-certs\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.351060 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.350930    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-envoy\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.351060 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.351020    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-socket\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.351216 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.351077    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-token\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.351216 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.351136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-credential-socket\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.351216 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.351171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-data\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.351216 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.351198    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-podinfo\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.351369 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.351232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9kr\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-kube-api-access-wk9kr\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.452419 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452352    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-data\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.452419 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-podinfo\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.452419 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9kr\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-kube-api-access-wk9kr\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.452703 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452444    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istiod-ca-cert\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.452703 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-certs\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.452703 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-envoy\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.452703 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-socket\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.452703 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-token\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.452703 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-credential-socket\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.453006 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452859    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-data\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.453006 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.452911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-certs\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.453131 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.453105    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-credential-socket\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.453171 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.453126    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-socket\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.453300 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.453281    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istiod-ca-cert\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.455052 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.455025    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-envoy\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.455146 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.455075    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-podinfo\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.460688 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.460666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-token\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.460792 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.460723    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9kr\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-kube-api-access-wk9kr\") pod \"e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") " pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.513785 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.513766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:53.652228 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.648716    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"]
Jul 13 21:38:53.655249 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:38:53.655221    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbdd5be_7293_434b_b403_cbbb1eed90cf.slice/crio-77c10831045e87544392edc7c1087585d9d1ee85f4f27a38dc64bf9b44fdb59a WatchSource:0}: Error finding container 77c10831045e87544392edc7c1087585d9d1ee85f4f27a38dc64bf9b44fdb59a: Status 404 returned error can't find the container with id 77c10831045e87544392edc7c1087585d9d1ee85f4f27a38dc64bf9b44fdb59a
Jul 13 21:38:53.657425 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.657392    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:38:53.657511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.657455    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:38:53.657511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.657481    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:38:53.851174 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.851131    2564 patch_prober.go:28] interesting pod/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 21:38:53.851310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:53.851195    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:38:54.330526 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.330482    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:54.387014 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.386982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" event={"ID":"0bbdd5be-7293-434b-b403-cbbb1eed90cf","Type":"ContainerStarted","Data":"78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7"}
Jul 13 21:38:54.387163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.387034    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" event={"ID":"0bbdd5be-7293-434b-b403-cbbb1eed90cf","Type":"ContainerStarted","Data":"77c10831045e87544392edc7c1087585d9d1ee85f4f27a38dc64bf9b44fdb59a"}
Jul 13 21:38:54.410539 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.410486    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" podStartSLOduration=1.410471375 podStartE2EDuration="1.410471375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:38:54.409107779 +0000 UTC m=+1781.066764787" watchObservedRunningTime="2026-07-13 21:38:54.410471375 +0000 UTC m=+1781.068128382"
Jul 13 21:38:54.415838 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.415808    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:54.417139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.417114    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"]
Jul 13 21:38:54.420543 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.420522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.422822 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.422792    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ae8a7501-serving-cert\""
Jul 13 21:38:54.429300 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.429248    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"]
Jul 13 21:38:54.486343 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.486315    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:54.514802 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.514775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:54.519503 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.519484    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:54.563609 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.563549    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0fc3c9a-4308-4324-b05a-845e334635d8-openshift-service-ca\") pod \"maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.563786 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.563652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0fc3c9a-4308-4324-b05a-845e334635d8-maas-api-tls\") pod \"maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.563786 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.563695    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfd7m\" (UniqueName: \"kubernetes.io/projected/f0fc3c9a-4308-4324-b05a-845e334635d8-kube-api-access-lfd7m\") pod \"maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.628046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.627970    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:54.664888 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.664862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0fc3c9a-4308-4324-b05a-845e334635d8-openshift-service-ca\") pod \"maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.664996 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.664905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0fc3c9a-4308-4324-b05a-845e334635d8-maas-api-tls\") pod \"maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.664996 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.664928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfd7m\" (UniqueName: \"kubernetes.io/projected/f0fc3c9a-4308-4324-b05a-845e334635d8-kube-api-access-lfd7m\") pod \"maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.665484 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.665461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0fc3c9a-4308-4324-b05a-845e334635d8-openshift-service-ca\") pod \"maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.667552 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.667521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0fc3c9a-4308-4324-b05a-845e334635d8-maas-api-tls\") pod \"maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.673659 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.673639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfd7m\" (UniqueName: \"kubernetes.io/projected/f0fc3c9a-4308-4324-b05a-845e334635d8-kube-api-access-lfd7m\") pod \"maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.732831 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.732809    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:54.865480 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:54.865455    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"]
Jul 13 21:38:54.867252 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:38:54.867226    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fc3c9a_4308_4324_b05a_845e334635d8.slice/crio-2613ac7332066b7d9cb7551e37f8f07058c348045969243aff1f3e5fbd910ef7 WatchSource:0}: Error finding container 2613ac7332066b7d9cb7551e37f8f07058c348045969243aff1f3e5fbd910ef7: Status 404 returned error can't find the container with id 2613ac7332066b7d9cb7551e37f8f07058c348045969243aff1f3e5fbd910ef7
Jul 13 21:38:55.105237 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:55.105204    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:55.212204 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:55.212164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:55.393878 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:55.393793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh" event={"ID":"f0fc3c9a-4308-4324-b05a-845e334635d8","Type":"ContainerStarted","Data":"d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced"}
Jul 13 21:38:55.394329 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:55.394035    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:55.394329 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:55.394076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:38:55.394329 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:55.394099    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh" event={"ID":"f0fc3c9a-4308-4324-b05a-845e334635d8","Type":"ContainerStarted","Data":"2613ac7332066b7d9cb7551e37f8f07058c348045969243aff1f3e5fbd910ef7"}
Jul 13 21:38:55.394795 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:55.394776    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:38:55.414938 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:55.414900    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh" podStartSLOduration=1.017198817 podStartE2EDuration="1.414886024s" totalImagesPullingTime="397.687207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:38:55.413063694 +0000 UTC m=+1782.070720701" watchObservedRunningTime="2026-07-13 21:38:55.414886024 +0000 UTC m=+1782.072543029"
Jul 13 21:38:58.189659 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.189633    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:38:58.294737 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.294705    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-credential-socket\") pod \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") "
Jul 13 21:38:58.294899 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.294755    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-envoy\") pod \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") "
Jul 13 21:38:58.294899 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.294774    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxgww\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-kube-api-access-nxgww\") pod \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") "
Jul 13 21:38:58.294899 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.294789    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-podinfo\") pod \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") "
Jul 13 21:38:58.294899 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.294807    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-data\") pod \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") "
Jul 13 21:38:58.294899 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.294839    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istiod-ca-cert\") pod \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") "
Jul 13 21:38:58.294899 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.294865    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-socket\") pod \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") "
Jul 13 21:38:58.295224 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.294915    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-certs\") pod \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") "
Jul 13 21:38:58.295224 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.294951    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-token\") pod \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\" (UID: \"d2c265e6-647d-4c8e-ad6c-92477f249e5c\") "
Jul 13 21:38:58.295224 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.295033    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2c265e6-647d-4c8e-ad6c-92477f249e5c" (UID: "d2c265e6-647d-4c8e-ad6c-92477f249e5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:38:58.295224 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.295200    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2c265e6-647d-4c8e-ad6c-92477f249e5c" (UID: "d2c265e6-647d-4c8e-ad6c-92477f249e5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:38:58.295440 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.295253    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2c265e6-647d-4c8e-ad6c-92477f249e5c" (UID: "d2c265e6-647d-4c8e-ad6c-92477f249e5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:38:58.295440 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.295263    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:58.295440 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.295303    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:58.295844 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.295813    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2c265e6-647d-4c8e-ad6c-92477f249e5c" (UID: "d2c265e6-647d-4c8e-ad6c-92477f249e5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:38:58.295940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.295839    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2c265e6-647d-4c8e-ad6c-92477f249e5c" (UID: "d2c265e6-647d-4c8e-ad6c-92477f249e5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:38:58.297124 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.297098    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-kube-api-access-nxgww" (OuterVolumeSpecName: "kube-api-access-nxgww") pod "d2c265e6-647d-4c8e-ad6c-92477f249e5c" (UID: "d2c265e6-647d-4c8e-ad6c-92477f249e5c"). InnerVolumeSpecName "kube-api-access-nxgww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:38:58.297124 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.297108    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2c265e6-647d-4c8e-ad6c-92477f249e5c" (UID: "d2c265e6-647d-4c8e-ad6c-92477f249e5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:38:58.297950 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.297922    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2c265e6-647d-4c8e-ad6c-92477f249e5c" (UID: "d2c265e6-647d-4c8e-ad6c-92477f249e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:38:58.298038 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.297953    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2c265e6-647d-4c8e-ad6c-92477f249e5c" (UID: "d2c265e6-647d-4c8e-ad6c-92477f249e5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:38:58.396043 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.395999    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:58.396043 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.396037    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxgww\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-kube-api-access-nxgww\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:58.396043 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.396051    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:58.396286 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.396064    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:58.396286 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.396075    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:58.396286 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.396086    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c265e6-647d-4c8e-ad6c-92477f249e5c-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:58.396286 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.396096    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c265e6-647d-4c8e-ad6c-92477f249e5c-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:38:58.412067 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.412042    2564 generic.go:356] "Generic (PLEG): container finished" podID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerID="997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1" exitCode=0
Jul 13 21:38:58.412197 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.412092    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"
Jul 13 21:38:58.412197 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.412096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" event={"ID":"d2c265e6-647d-4c8e-ad6c-92477f249e5c","Type":"ContainerDied","Data":"997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1"}
Jul 13 21:38:58.412269 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.412198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r" event={"ID":"d2c265e6-647d-4c8e-ad6c-92477f249e5c","Type":"ContainerDied","Data":"6a16a27490d0d07415dd1e997d0873749ee06bd7e9ae2c31af9a0e2d5d0d1c3f"}
Jul 13 21:38:58.412269 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.412220    2564 scope.go:117] "RemoveContainer" containerID="997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1"
Jul 13 21:38:58.421796 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.421778    2564 scope.go:117] "RemoveContainer" containerID="8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b"
Jul 13 21:38:58.429508 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.429492    2564 scope.go:117] "RemoveContainer" containerID="997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1"
Jul 13 21:38:58.429799 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:38:58.429780    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1\": container with ID starting with 997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1 not found: ID does not exist" containerID="997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1"
Jul 13 21:38:58.429864 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.429805    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1"} err="failed to get container status \"997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1\": rpc error: code = NotFound desc = could not find container \"997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1\": container with ID starting with 997163ee7bd41d2624def48cb68d117b661033a2de710d36caf172eb7e807ea1 not found: ID does not exist"
Jul 13 21:38:58.429864 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.429824    2564 scope.go:117] "RemoveContainer" containerID="8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b"
Jul 13 21:38:58.430049 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:38:58.430035    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b\": container with ID starting with 8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b not found: ID does not exist" containerID="8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b"
Jul 13 21:38:58.430095 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.430053    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b"} err="failed to get container status \"8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b\": rpc error: code = NotFound desc = could not find container \"8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b\": container with ID starting with 8d22f02dff249d7959bc9ec93c10f47ce35909d19f6cfc403fdc40e76c96c68b not found: ID does not exist"
Jul 13 21:38:58.435200 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.435180    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"]
Jul 13 21:38:58.439411 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.439392    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-057af81a-openshift-default-864467b9fc-mwp8r"]
Jul 13 21:38:58.713225 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.713145    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"]
Jul 13 21:38:58.713472 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.713459    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy"
Jul 13 21:38:58.713514 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.713473    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy"
Jul 13 21:38:58.713514 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.713483    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy"
Jul 13 21:38:58.713514 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.713488    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy"
Jul 13 21:38:58.713629 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.713544    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy"
Jul 13 21:38:58.713629 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.713552    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" containerName="istio-proxy"
Jul 13 21:38:58.717887 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.717871    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"
Jul 13 21:38:58.720350 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.720319    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:38:58.726168 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.726140    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"]
Jul 13 21:38:58.800295 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.800261    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd3bc3a-a64a-483e-98dc-1773716e4a3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6\" (UID: \"afd3bc3a-a64a-483e-98dc-1773716e4a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"
Jul 13 21:38:58.901301 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.901266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd3bc3a-a64a-483e-98dc-1773716e4a3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6\" (UID: \"afd3bc3a-a64a-483e-98dc-1773716e4a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"
Jul 13 21:38:58.901893 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:58.901874    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd3bc3a-a64a-483e-98dc-1773716e4a3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6\" (UID: \"afd3bc3a-a64a-483e-98dc-1773716e4a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"
Jul 13 21:38:59.028882 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:59.028858    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"
Jul 13 21:38:59.212804 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:59.212722    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"]
Jul 13 21:38:59.216176 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:38:59.216144    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd3bc3a_a64a_483e_98dc_1773716e4a3b.slice/crio-0b125597e79a53047a73499837a9e13d83d2ebdf390e671175ff331e3e2294d3 WatchSource:0}: Error finding container 0b125597e79a53047a73499837a9e13d83d2ebdf390e671175ff331e3e2294d3: Status 404 returned error can't find the container with id 0b125597e79a53047a73499837a9e13d83d2ebdf390e671175ff331e3e2294d3
Jul 13 21:38:59.419639 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:59.419541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6" event={"ID":"afd3bc3a-a64a-483e-98dc-1773716e4a3b","Type":"ContainerStarted","Data":"0b125597e79a53047a73499837a9e13d83d2ebdf390e671175ff331e3e2294d3"}
Jul 13 21:38:59.821225 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:38:59.821191    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c265e6-647d-4c8e-ad6c-92477f249e5c" path="/var/lib/kubelet/pods/d2c265e6-647d-4c8e-ad6c-92477f249e5c/volumes"
Jul 13 21:39:00.426231 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:00.426137    2564 generic.go:356] "Generic (PLEG): container finished" podID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerID="7aee9264f26670e7bab037621428768236f8fe275450e5c16b06146b72a18c48" exitCode=7
Jul 13 21:39:00.426231 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:00.426183    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6" event={"ID":"afd3bc3a-a64a-483e-98dc-1773716e4a3b","Type":"ContainerDied","Data":"7aee9264f26670e7bab037621428768236f8fe275450e5c16b06146b72a18c48"}
Jul 13 21:39:00.426747 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:00.426420    2564 scope.go:117] "RemoveContainer" containerID="7aee9264f26670e7bab037621428768236f8fe275450e5c16b06146b72a18c48"
Jul 13 21:39:01.404212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:01.404191    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:39:01.433523 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:01.433496    2564 generic.go:356] "Generic (PLEG): container finished" podID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerID="5c3c13658c837d5ad564c0491ba5f9d8675e1da9dad19e598a83346c7fb3eaa7" exitCode=7
Jul 13 21:39:01.433903 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:01.433544    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6" event={"ID":"afd3bc3a-a64a-483e-98dc-1773716e4a3b","Type":"ContainerDied","Data":"5c3c13658c837d5ad564c0491ba5f9d8675e1da9dad19e598a83346c7fb3eaa7"}
Jul 13 21:39:01.433903 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:01.433624    2564 scope.go:117] "RemoveContainer" containerID="7aee9264f26670e7bab037621428768236f8fe275450e5c16b06146b72a18c48"
Jul 13 21:39:01.433903 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:01.433808    2564 scope.go:117] "RemoveContainer" containerID="5c3c13658c837d5ad564c0491ba5f9d8675e1da9dad19e598a83346c7fb3eaa7"
Jul 13 21:39:01.434067 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:39:01.434013    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6_odh-ai-gateway-infra(afd3bc3a-a64a-483e-98dc-1773716e4a3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b"
Jul 13 21:39:02.440393 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:02.440312    2564 scope.go:117] "RemoveContainer" containerID="5c3c13658c837d5ad564c0491ba5f9d8675e1da9dad19e598a83346c7fb3eaa7"
Jul 13 21:39:02.440752 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:39:02.440486    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6_odh-ai-gateway-infra(afd3bc3a-a64a-483e-98dc-1773716e4a3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b"
Jul 13 21:39:04.784866 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:04.784835    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:04.886154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:04.886127    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:05.456049 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:05.456018    2564 generic.go:356] "Generic (PLEG): container finished" podID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerID="78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7" exitCode=0
Jul 13 21:39:05.456247 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:05.456098    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" event={"ID":"0bbdd5be-7293-434b-b403-cbbb1eed90cf","Type":"ContainerDied","Data":"78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7"}
Jul 13 21:39:05.456508 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:05.456492    2564 scope.go:117] "RemoveContainer" containerID="78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7"
Jul 13 21:39:05.457488 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:05.457463    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:39:05.457547 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:05.457528    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:39:05.457618 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:05.457566    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:39:06.465992 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:06.465956    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" event={"ID":"0bbdd5be-7293-434b-b403-cbbb1eed90cf","Type":"ContainerStarted","Data":"11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384"}
Jul 13 21:39:06.514116 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:06.514084    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:39:06.515300 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:06.515275    2564 patch_prober.go:28] interesting pod/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 13 21:39:06.515429 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:06.515335    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 13 21:39:07.514472 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:07.514442    2564 patch_prober.go:28] interesting pod/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 13 21:39:07.514884 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:07.514495    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 13 21:39:08.282074 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:08.282016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:08.514118 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:08.514082    2564 patch_prober.go:28] interesting pod/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 13 21:39:08.514279 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:08.514150    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 13 21:39:09.515165 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:09.515134    2564 patch_prober.go:28] interesting pod/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 13 21:39:09.515663 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:09.515225    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 13 21:39:10.394517 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:10.394486    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:39:10.519029 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:10.519000    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:39:10.519988 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:10.519969    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:39:13.897302 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:13.897275    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:39:13.903192 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:13.903173    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:39:14.816925 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:14.816893    2564 scope.go:117] "RemoveContainer" containerID="5c3c13658c837d5ad564c0491ba5f9d8675e1da9dad19e598a83346c7fb3eaa7"
Jul 13 21:39:15.514954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:15.514921    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6" event={"ID":"afd3bc3a-a64a-483e-98dc-1773716e4a3b","Type":"ContainerStarted","Data":"c5c0b6586edef4fa8de1e5793d0e15cd308f48d6890c73908372792df7ad40bf"}
Jul 13 21:39:15.534757 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:15.534717    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6" podStartSLOduration=17.53470296 podStartE2EDuration="17.53470296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:39:15.533041569 +0000 UTC m=+1802.190698575" watchObservedRunningTime="2026-07-13 21:39:15.53470296 +0000 UTC m=+1802.192359966"
Jul 13 21:39:16.521355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:16.521323    2564 generic.go:356] "Generic (PLEG): container finished" podID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerID="c5c0b6586edef4fa8de1e5793d0e15cd308f48d6890c73908372792df7ad40bf" exitCode=0
Jul 13 21:39:16.521790 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:16.521394    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6" event={"ID":"afd3bc3a-a64a-483e-98dc-1773716e4a3b","Type":"ContainerDied","Data":"c5c0b6586edef4fa8de1e5793d0e15cd308f48d6890c73908372792df7ad40bf"}
Jul 13 21:39:16.521790 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:16.521452    2564 scope.go:117] "RemoveContainer" containerID="5c3c13658c837d5ad564c0491ba5f9d8675e1da9dad19e598a83346c7fb3eaa7"
Jul 13 21:39:16.547763 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:16.547738    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"]
Jul 13 21:39:17.659755 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:17.659729    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"
Jul 13 21:39:17.762199 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:17.762174    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd3bc3a-a64a-483e-98dc-1773716e4a3b-maas-api-service-ca\") pod \"afd3bc3a-a64a-483e-98dc-1773716e4a3b\" (UID: \"afd3bc3a-a64a-483e-98dc-1773716e4a3b\") "
Jul 13 21:39:17.762482 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:17.762463    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd3bc3a-a64a-483e-98dc-1773716e4a3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "afd3bc3a-a64a-483e-98dc-1773716e4a3b" (UID: "afd3bc3a-a64a-483e-98dc-1773716e4a3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:39:17.863508 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:17.863483    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd3bc3a-a64a-483e-98dc-1773716e4a3b-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:39:18.534254 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:18.534215    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6" event={"ID":"afd3bc3a-a64a-483e-98dc-1773716e4a3b","Type":"ContainerDied","Data":"0b125597e79a53047a73499837a9e13d83d2ebdf390e671175ff331e3e2294d3"}
Jul 13 21:39:18.534254 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:18.534242    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"
Jul 13 21:39:18.534527 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:18.534275    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b125597e79a53047a73499837a9e13d83d2ebdf390e671175ff331e3e2294d3"
Jul 13 21:39:18.563248 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:18.563225    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"]
Jul 13 21:39:18.568842 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:18.568821    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-zrjl6"]
Jul 13 21:39:19.822823 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:19.822782    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" path="/var/lib/kubelet/pods/afd3bc3a-a64a-483e-98dc-1773716e4a3b/volumes"
Jul 13 21:39:48.754380 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.754340    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"]
Jul 13 21:39:48.754927 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.754715    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerName="revoke-keys"
Jul 13 21:39:48.754927 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.754728    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerName="revoke-keys"
Jul 13 21:39:48.754927 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.754754    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerName="revoke-keys"
Jul 13 21:39:48.754927 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.754764    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerName="revoke-keys"
Jul 13 21:39:48.754927 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.754816    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerName="revoke-keys"
Jul 13 21:39:48.754927 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.754824    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerName="revoke-keys"
Jul 13 21:39:48.756834 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.756817    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"
Jul 13 21:39:48.759362 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.759339    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:39:48.765681 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.765644    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"]
Jul 13 21:39:48.887915 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.887889    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0eb5b7-2c49-4df7-ae24-9808bc5498f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz\" (UID: \"ee0eb5b7-2c49-4df7-ae24-9808bc5498f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"
Jul 13 21:39:48.989243 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.989218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0eb5b7-2c49-4df7-ae24-9808bc5498f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz\" (UID: \"ee0eb5b7-2c49-4df7-ae24-9808bc5498f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"
Jul 13 21:39:48.989782 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:48.989766    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0eb5b7-2c49-4df7-ae24-9808bc5498f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz\" (UID: \"ee0eb5b7-2c49-4df7-ae24-9808bc5498f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"
Jul 13 21:39:49.069382 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:49.069335    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"
Jul 13 21:39:49.202043 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:49.202021    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"]
Jul 13 21:39:49.204064 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:39:49.204026    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0eb5b7_2c49_4df7_ae24_9808bc5498f8.slice/crio-931634587c8e71e9dd395b96089b334d8aa307fdfc62db70f634121cdf94005b WatchSource:0}: Error finding container 931634587c8e71e9dd395b96089b334d8aa307fdfc62db70f634121cdf94005b: Status 404 returned error can't find the container with id 931634587c8e71e9dd395b96089b334d8aa307fdfc62db70f634121cdf94005b
Jul 13 21:39:49.692178 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:49.692086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz" event={"ID":"ee0eb5b7-2c49-4df7-ae24-9808bc5498f8","Type":"ContainerStarted","Data":"d449299823cb0825ec635423d50b1a75c4a93e7742541296dc430c843b6d4855"}
Jul 13 21:39:49.692178 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:49.692136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz" event={"ID":"ee0eb5b7-2c49-4df7-ae24-9808bc5498f8","Type":"ContainerStarted","Data":"931634587c8e71e9dd395b96089b334d8aa307fdfc62db70f634121cdf94005b"}
Jul 13 21:39:49.710648 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:49.710575    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz" podStartSLOduration=1.71055997 podStartE2EDuration="1.71055997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:39:49.708472525 +0000 UTC m=+1836.366129535" watchObservedRunningTime="2026-07-13 21:39:49.71055997 +0000 UTC m=+1836.368216976"
Jul 13 21:39:50.697912 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:50.697877    2564 generic.go:356] "Generic (PLEG): container finished" podID="ee0eb5b7-2c49-4df7-ae24-9808bc5498f8" containerID="d449299823cb0825ec635423d50b1a75c4a93e7742541296dc430c843b6d4855" exitCode=0
Jul 13 21:39:50.698322 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:50.698005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz" event={"ID":"ee0eb5b7-2c49-4df7-ae24-9808bc5498f8","Type":"ContainerDied","Data":"d449299823cb0825ec635423d50b1a75c4a93e7742541296dc430c843b6d4855"}
Jul 13 21:39:51.855147 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:51.855126    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"
Jul 13 21:39:51.912897 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:51.912865    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0eb5b7-2c49-4df7-ae24-9808bc5498f8-maas-api-service-ca\") pod \"ee0eb5b7-2c49-4df7-ae24-9808bc5498f8\" (UID: \"ee0eb5b7-2c49-4df7-ae24-9808bc5498f8\") "
Jul 13 21:39:51.913170 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:51.913150    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0eb5b7-2c49-4df7-ae24-9808bc5498f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee0eb5b7-2c49-4df7-ae24-9808bc5498f8" (UID: "ee0eb5b7-2c49-4df7-ae24-9808bc5498f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:39:52.013558 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:52.013534    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0eb5b7-2c49-4df7-ae24-9808bc5498f8-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:39:52.708953 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:52.708913    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz"
Jul 13 21:39:52.709155 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:52.708953    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz" event={"ID":"ee0eb5b7-2c49-4df7-ae24-9808bc5498f8","Type":"ContainerDied","Data":"931634587c8e71e9dd395b96089b334d8aa307fdfc62db70f634121cdf94005b"}
Jul 13 21:39:52.709155 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:52.709041    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="931634587c8e71e9dd395b96089b334d8aa307fdfc62db70f634121cdf94005b"
Jul 13 21:39:54.170361 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:54.170318    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:59.149241 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.149206    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"]
Jul 13 21:39:59.149715 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.149440    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh" podUID="f0fc3c9a-4308-4324-b05a-845e334635d8" containerName="maas-api" containerID="cri-o://d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced" gracePeriod=30
Jul 13 21:39:59.177275 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.177251    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:59.391336 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.391313    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:39:59.474193 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.474138    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfd7m\" (UniqueName: \"kubernetes.io/projected/f0fc3c9a-4308-4324-b05a-845e334635d8-kube-api-access-lfd7m\") pod \"f0fc3c9a-4308-4324-b05a-845e334635d8\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") "
Jul 13 21:39:59.474193 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.474185    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0fc3c9a-4308-4324-b05a-845e334635d8-openshift-service-ca\") pod \"f0fc3c9a-4308-4324-b05a-845e334635d8\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") "
Jul 13 21:39:59.474360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.474205    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0fc3c9a-4308-4324-b05a-845e334635d8-maas-api-tls\") pod \"f0fc3c9a-4308-4324-b05a-845e334635d8\" (UID: \"f0fc3c9a-4308-4324-b05a-845e334635d8\") "
Jul 13 21:39:59.474468 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.474447    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0fc3c9a-4308-4324-b05a-845e334635d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f0fc3c9a-4308-4324-b05a-845e334635d8" (UID: "f0fc3c9a-4308-4324-b05a-845e334635d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:39:59.476338 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.476313    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0fc3c9a-4308-4324-b05a-845e334635d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f0fc3c9a-4308-4324-b05a-845e334635d8" (UID: "f0fc3c9a-4308-4324-b05a-845e334635d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:39:59.476431 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.476385    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fc3c9a-4308-4324-b05a-845e334635d8-kube-api-access-lfd7m" (OuterVolumeSpecName: "kube-api-access-lfd7m") pod "f0fc3c9a-4308-4324-b05a-845e334635d8" (UID: "f0fc3c9a-4308-4324-b05a-845e334635d8"). InnerVolumeSpecName "kube-api-access-lfd7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:39:59.575460 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.575436    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfd7m\" (UniqueName: \"kubernetes.io/projected/f0fc3c9a-4308-4324-b05a-845e334635d8-kube-api-access-lfd7m\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:39:59.575460 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.575458    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0fc3c9a-4308-4324-b05a-845e334635d8-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:39:59.575611 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.575471    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0fc3c9a-4308-4324-b05a-845e334635d8-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:39:59.741525 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.741492    2564 generic.go:356] "Generic (PLEG): container finished" podID="f0fc3c9a-4308-4324-b05a-845e334635d8" containerID="d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced" exitCode=0
Jul 13 21:39:59.741695 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.741537    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"
Jul 13 21:39:59.741695 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.741606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh" event={"ID":"f0fc3c9a-4308-4324-b05a-845e334635d8","Type":"ContainerDied","Data":"d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced"}
Jul 13 21:39:59.741695 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.741668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh" event={"ID":"f0fc3c9a-4308-4324-b05a-845e334635d8","Type":"ContainerDied","Data":"2613ac7332066b7d9cb7551e37f8f07058c348045969243aff1f3e5fbd910ef7"}
Jul 13 21:39:59.741695 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.741694    2564 scope.go:117] "RemoveContainer" containerID="d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced"
Jul 13 21:39:59.750486 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.750472    2564 scope.go:117] "RemoveContainer" containerID="d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced"
Jul 13 21:39:59.750745 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:39:59.750727    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced\": container with ID starting with d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced not found: ID does not exist" containerID="d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced"
Jul 13 21:39:59.750793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.750752    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced"} err="failed to get container status \"d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced\": rpc error: code = NotFound desc = could not find container \"d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced\": container with ID starting with d408ee0cce04c8a35034278018739b4b872d4e46dfa62db0e46f9359a122fced not found: ID does not exist"
Jul 13 21:39:59.762514 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.762496    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"]
Jul 13 21:39:59.766095 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.766076    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ae8a7501-6cb7f86b58-nwrbh"]
Jul 13 21:39:59.826757 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:39:59.826731    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0fc3c9a-4308-4324-b05a-845e334635d8" path="/var/lib/kubelet/pods/f0fc3c9a-4308-4324-b05a-845e334635d8/volumes"
Jul 13 21:40:03.859961 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:03.859925    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:09.458294 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:09.458255    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:09.558772 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:09.558738    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:13.997374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:13.997341    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"]
Jul 13 21:40:13.997813 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:13.997611    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy" containerID="cri-o://11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384" gracePeriod=30
Jul 13 21:40:14.023959 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:14.023931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:14.582601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:14.582531    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:14.677104 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:14.677078    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:14.778986 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:14.778955    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:16.242546 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:16.242505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:19.253851 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.253828    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:40:19.423827 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.423800    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-credential-socket\") pod \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") "
Jul 13 21:40:19.423980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.423842    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-socket\") pod \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") "
Jul 13 21:40:19.423980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.423890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-token\") pod \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") "
Jul 13 21:40:19.423980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.423913    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istiod-ca-cert\") pod \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") "
Jul 13 21:40:19.423980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.423927    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-certs\") pod \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") "
Jul 13 21:40:19.423980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.423943    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk9kr\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-kube-api-access-wk9kr\") pod \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") "
Jul 13 21:40:19.423980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.423969    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-envoy\") pod \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") "
Jul 13 21:40:19.424288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.423989    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-data\") pod \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") "
Jul 13 21:40:19.424288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.424014    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-podinfo\") pod \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\" (UID: \"0bbdd5be-7293-434b-b403-cbbb1eed90cf\") "
Jul 13 21:40:19.424288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.424062    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0bbdd5be-7293-434b-b403-cbbb1eed90cf" (UID: "0bbdd5be-7293-434b-b403-cbbb1eed90cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:40:19.424288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.424181    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0bbdd5be-7293-434b-b403-cbbb1eed90cf" (UID: "0bbdd5be-7293-434b-b403-cbbb1eed90cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:40:19.424288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.424249    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:40:19.424288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.424267    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:40:19.424288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.424265    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0bbdd5be-7293-434b-b403-cbbb1eed90cf" (UID: "0bbdd5be-7293-434b-b403-cbbb1eed90cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:40:19.424846 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.424491    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0bbdd5be-7293-434b-b403-cbbb1eed90cf" (UID: "0bbdd5be-7293-434b-b403-cbbb1eed90cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:40:19.425189 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.425163    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "0bbdd5be-7293-434b-b403-cbbb1eed90cf" (UID: "0bbdd5be-7293-434b-b403-cbbb1eed90cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:40:19.426372 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.426342    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0bbdd5be-7293-434b-b403-cbbb1eed90cf" (UID: "0bbdd5be-7293-434b-b403-cbbb1eed90cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:40:19.426481 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.426407    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "0bbdd5be-7293-434b-b403-cbbb1eed90cf" (UID: "0bbdd5be-7293-434b-b403-cbbb1eed90cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:40:19.426481 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.426428    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-kube-api-access-wk9kr" (OuterVolumeSpecName: "kube-api-access-wk9kr") pod "0bbdd5be-7293-434b-b403-cbbb1eed90cf" (UID: "0bbdd5be-7293-434b-b403-cbbb1eed90cf"). InnerVolumeSpecName "kube-api-access-wk9kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:40:19.426481 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.426439    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0bbdd5be-7293-434b-b403-cbbb1eed90cf" (UID: "0bbdd5be-7293-434b-b403-cbbb1eed90cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:40:19.525295 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.525274    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:40:19.525295 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.525293    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:40:19.525417 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.525303    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:40:19.525417 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.525311    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk9kr\" (UniqueName: \"kubernetes.io/projected/0bbdd5be-7293-434b-b403-cbbb1eed90cf-kube-api-access-wk9kr\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:40:19.525417 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.525320    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:40:19.525417 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.525328    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:40:19.525417 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.525335    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbdd5be-7293-434b-b403-cbbb1eed90cf-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:40:19.842991 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.842965    2564 generic.go:356] "Generic (PLEG): container finished" podID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerID="11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384" exitCode=0
Jul 13 21:40:19.843113 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.843007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" event={"ID":"0bbdd5be-7293-434b-b403-cbbb1eed90cf","Type":"ContainerDied","Data":"11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384"}
Jul 13 21:40:19.843113 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.843027    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"
Jul 13 21:40:19.843113 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.843066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd" event={"ID":"0bbdd5be-7293-434b-b403-cbbb1eed90cf","Type":"ContainerDied","Data":"77c10831045e87544392edc7c1087585d9d1ee85f4f27a38dc64bf9b44fdb59a"}
Jul 13 21:40:19.843113 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.843088    2564 scope.go:117] "RemoveContainer" containerID="11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384"
Jul 13 21:40:19.852409 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.852391    2564 scope.go:117] "RemoveContainer" containerID="78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7"
Jul 13 21:40:19.859991 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.859970    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"]
Jul 13 21:40:19.862958 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.862941    2564 scope.go:117] "RemoveContainer" containerID="11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384"
Jul 13 21:40:19.864139 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:40:19.864012    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384\": container with ID starting with 11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384 not found: ID does not exist" containerID="11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384"
Jul 13 21:40:19.864139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.864050    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384"} err="failed to get container status \"11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384\": rpc error: code = NotFound desc = could not find container \"11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384\": container with ID starting with 11f53d3f65628449f2afb319a7716a389fa58170a9501ec706b0e80ebb893384 not found: ID does not exist"
Jul 13 21:40:19.864139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.864072    2564 scope.go:117] "RemoveContainer" containerID="78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7"
Jul 13 21:40:19.865149 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:40:19.865117    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7\": container with ID starting with 78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7 not found: ID does not exist" containerID="78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7"
Jul 13 21:40:19.865274 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.865158    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7"} err="failed to get container status \"78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7\": rpc error: code = NotFound desc = could not find container \"78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7\": container with ID starting with 78a408be6099b89e8b90ffa76f61b0dfc46d28af09a56046b03d9716652ce0a7 not found: ID does not exist"
Jul 13 21:40:19.865665 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:19.865634    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ae8a7501-openshift-default-75bd7b4cc9-tj4wd"]
Jul 13 21:40:20.337357 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:20.337324    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:21.821572 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:21.821543    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" path="/var/lib/kubelet/pods/0bbdd5be-7293-434b-b403-cbbb1eed90cf/volumes"
Jul 13 21:40:22.191028 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:22.190941    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:22.256786 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:22.256758    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:22.347956 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:22.347926    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:22.447384 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:22.447303    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:22.547668 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:22.547641    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:22.900277 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:22.900250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:37.756854 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:37.756759    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:48.918537 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:48.918497    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:54.611486 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:54.611412    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:54.674664 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:54.674630    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:54.772570 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:54.772541    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:54.877327 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:40:54.877252    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:10.695325 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:10.695288    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:13.975343 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:13.975313    2564 scope.go:117] "RemoveContainer" containerID="0428dd2760f7fd0e76e116e58181a3bbd507441ff2fa451f70a1d40ee06a400e"
Jul 13 21:41:16.373807 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:16.373762    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:17.629592 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:17.629552    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:27.095329 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:27.095286    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:27.939891 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:27.939855    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:33.728675 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:33.728637    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:33.796775 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:33.796743    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:33.879201 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:33.879167    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:33.977711 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:33.977683    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:34.078722 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:34.078691    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:34.784765 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:34.784720    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:34.887373 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:34.887341    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:34.990332 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:34.990303    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:36.504119 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:36.504086    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:42.091965 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:42.091928    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:42.152243 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:42.152187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:42.753145 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:42.753117    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:48.389065 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.389027    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:48.487788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.487758    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:48.634749 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.634721    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:48.898757 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.898727    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"]
Jul 13 21:41:48.899017 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899003    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy"
Jul 13 21:41:48.899017 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899017    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899029    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899035    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899046    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0fc3c9a-4308-4324-b05a-845e334635d8" containerName="maas-api"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899051    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0fc3c9a-4308-4324-b05a-845e334635d8" containerName="maas-api"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899065    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerName="revoke-keys"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899069    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerName="revoke-keys"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899077    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0eb5b7-2c49-4df7-ae24-9808bc5498f8" containerName="revoke-keys"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899082    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0eb5b7-2c49-4df7-ae24-9808bc5498f8" containerName="revoke-keys"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899132    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0fc3c9a-4308-4324-b05a-845e334635d8" containerName="maas-api"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899140    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd3bc3a-a64a-483e-98dc-1773716e4a3b" containerName="revoke-keys"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899146    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899151    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0eb5b7-2c49-4df7-ae24-9808bc5498f8" containerName="revoke-keys"
Jul 13 21:41:48.899158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.899157    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bbdd5be-7293-434b-b403-cbbb1eed90cf" containerName="istio-proxy"
Jul 13 21:41:48.902034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.902019    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:48.905115 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.905092    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8603bf9d-openshift-default-dockercfg-tprwb\""
Jul 13 21:41:48.935252 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.935233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"]
Jul 13 21:41:48.944952 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.944922    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:48.975037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.975013    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-certs\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:48.975143 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.975046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729c9da-a627-4131-8bf2-b839b3a1afdb-istiod-ca-cert\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:48.975143 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.975067    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-data\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:48.975143 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.975087    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8scnl\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-kube-api-access-8scnl\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:48.975143 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.975138    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-podinfo\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:48.975342 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.975164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-socket\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:48.975342 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.975240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-token\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:48.975342 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.975297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-credential-socket\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:48.975452 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:48.975340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-envoy\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.076632 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.076603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729c9da-a627-4131-8bf2-b839b3a1afdb-istiod-ca-cert\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.076632 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.076634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-data\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.076765 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.076746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8scnl\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-kube-api-access-8scnl\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.076801 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.076778    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-podinfo\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.076839 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.076799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-socket\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.076839 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.076823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-token\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.076930 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.076862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-credential-socket\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.076930 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.076896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-envoy\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.077024 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.076955    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-certs\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.077094 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.077065    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-data\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.077199 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.077173    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-socket\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.077311 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.077279    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-credential-socket\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.077439 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.077310    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-certs\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.077439 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.077373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729c9da-a627-4131-8bf2-b839b3a1afdb-istiod-ca-cert\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.079253 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.079234    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-envoy\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.079492 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.079475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-podinfo\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.085065 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.085033    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8scnl\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-kube-api-access-8scnl\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.085220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.085197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-token\") pod \"e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") " pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.086254 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.086227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:49.211396 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.211323    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:49.343230 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.343204    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"]
Jul 13 21:41:49.346769 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:41:49.346735    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd729c9da_a627_4131_8bf2_b839b3a1afdb.slice/crio-238073d48e2046f0ebc7d51479c457c435aff74e2b15b2ef7cea0b08ecaf2e85 WatchSource:0}: Error finding container 238073d48e2046f0ebc7d51479c457c435aff74e2b15b2ef7cea0b08ecaf2e85: Status 404 returned error can't find the container with id 238073d48e2046f0ebc7d51479c457c435aff74e2b15b2ef7cea0b08ecaf2e85
Jul 13 21:41:49.348858 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.348825    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:41:49.348972 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.348889    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:41:49.348972 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:49.348919    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:41:50.252033 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:50.251994    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" event={"ID":"d729c9da-a627-4131-8bf2-b839b3a1afdb","Type":"ContainerStarted","Data":"507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18"}
Jul 13 21:41:50.252385 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:50.252054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" event={"ID":"d729c9da-a627-4131-8bf2-b839b3a1afdb","Type":"ContainerStarted","Data":"238073d48e2046f0ebc7d51479c457c435aff74e2b15b2ef7cea0b08ecaf2e85"}
Jul 13 21:41:50.274132 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:50.274088    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" podStartSLOduration=2.274072703 podStartE2EDuration="2.274072703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:41:50.272758398 +0000 UTC m=+1956.930415431" watchObservedRunningTime="2026-07-13 21:41:50.274072703 +0000 UTC m=+1956.931729706"
Jul 13 21:41:51.212383 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:51.212345    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:51.216709 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:51.216685    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:51.256231 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:51.256203    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:51.256883 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:51.256867    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:41:52.129146 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.129102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:52.208146 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.208103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:52.209002 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.208982    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"]
Jul 13 21:41:52.212310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.212290    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.214794 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.214775    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8603bf9d-serving-cert\""
Jul 13 21:41:52.222480 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.222458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"]
Jul 13 21:41:52.276957 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.276926    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:52.302341 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.302095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmrw\" (UniqueName: \"kubernetes.io/projected/8c0e0510-697e-4b90-b94a-cccdf6a30059-kube-api-access-dhmrw\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.302341 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.302189    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0510-697e-4b90-b94a-cccdf6a30059-maas-api-tls\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.302522 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.302449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0e0510-697e-4b90-b94a-cccdf6a30059-openshift-service-ca\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.403661 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.403541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0510-697e-4b90-b94a-cccdf6a30059-maas-api-tls\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.403835 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:41:52.403688    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-serving-cert: secret "maas-api-e2e-mt-8603bf9d-serving-cert" not found
Jul 13 21:41:52.403835 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:41:52.403750    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c0e0510-697e-4b90-b94a-cccdf6a30059-maas-api-tls podName:8c0e0510-697e-4b90-b94a-cccdf6a30059 nodeName:}" failed. No retries permitted until 2026-07-13 21:41:52.903733635 +0000 UTC m=+1959.561390619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c0e0510-697e-4b90-b94a-cccdf6a30059-maas-api-tls") pod "maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp" (UID: "8c0e0510-697e-4b90-b94a-cccdf6a30059") : secret "maas-api-e2e-mt-8603bf9d-serving-cert" not found
Jul 13 21:41:52.403835 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.403765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0e0510-697e-4b90-b94a-cccdf6a30059-openshift-service-ca\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.403835 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.403828    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmrw\" (UniqueName: \"kubernetes.io/projected/8c0e0510-697e-4b90-b94a-cccdf6a30059-kube-api-access-dhmrw\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.404474 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.404454    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0e0510-697e-4b90-b94a-cccdf6a30059-openshift-service-ca\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.412924 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.412894    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmrw\" (UniqueName: \"kubernetes.io/projected/8c0e0510-697e-4b90-b94a-cccdf6a30059-kube-api-access-dhmrw\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.540512 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.540471    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:52.801061 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.801031    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:52.901412 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.901379    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:52.908741 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.908719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0510-697e-4b90-b94a-cccdf6a30059-maas-api-tls\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:52.911251 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:52.911227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0510-697e-4b90-b94a-cccdf6a30059-maas-api-tls\") pod \"maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:53.130434 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:53.130336    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:53.273897 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:53.273870    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"]
Jul 13 21:41:53.275739 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:41:53.275711    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0e0510_697e_4b90_b94a_cccdf6a30059.slice/crio-6f76134d603210edc663ee4e201c7bd7a968348367202f6df2960332a656ce32 WatchSource:0}: Error finding container 6f76134d603210edc663ee4e201c7bd7a968348367202f6df2960332a656ce32: Status 404 returned error can't find the container with id 6f76134d603210edc663ee4e201c7bd7a968348367202f6df2960332a656ce32
Jul 13 21:41:53.276969 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:53.276951    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:41:54.271059 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:54.271024    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp" event={"ID":"8c0e0510-697e-4b90-b94a-cccdf6a30059","Type":"ContainerStarted","Data":"f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102"}
Jul 13 21:41:54.271500 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:54.271162    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:41:54.271500 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:54.271220    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp" event={"ID":"8c0e0510-697e-4b90-b94a-cccdf6a30059","Type":"ContainerStarted","Data":"6f76134d603210edc663ee4e201c7bd7a968348367202f6df2960332a656ce32"}
Jul 13 21:41:54.287958 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:54.287919    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp" podStartSLOduration=1.6107212720000001 podStartE2EDuration="2.287904009s" totalImagesPullingTime="677.182737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:41:54.286259745 +0000 UTC m=+1960.943916752" watchObservedRunningTime="2026-07-13 21:41:54.287904009 +0000 UTC m=+1960.945561048"
Jul 13 21:41:56.605490 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:56.605452    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:57.215342 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:57.215302    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:57.373825 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:57.373789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:57.662561 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:41:57.662521    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:00.281191 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:00.281160    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:42:03.297529 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.297483    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:03.360835 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.360800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:03.661128 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.661018    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"]
Jul 13 21:42:03.665294 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.665267    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"
Jul 13 21:42:03.667831 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.667805    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:42:03.671733 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.671691    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"]
Jul 13 21:42:03.700518 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.700487    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b802c1-0996-4c3a-92b1-03c96f602cb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx\" (UID: \"10b802c1-0996-4c3a-92b1-03c96f602cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"
Jul 13 21:42:03.801488 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.801453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b802c1-0996-4c3a-92b1-03c96f602cb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx\" (UID: \"10b802c1-0996-4c3a-92b1-03c96f602cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"
Jul 13 21:42:03.802177 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.802156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b802c1-0996-4c3a-92b1-03c96f602cb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx\" (UID: \"10b802c1-0996-4c3a-92b1-03c96f602cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"
Jul 13 21:42:03.978755 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:03.978673    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"
Jul 13 21:42:04.123670 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:04.123565    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"]
Jul 13 21:42:04.126333 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:42:04.126304    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b802c1_0996_4c3a_92b1_03c96f602cb7.slice/crio-1740ea01801f15d32990509c36ba32272ff25f243a53d08a190b299ddc413d83 WatchSource:0}: Error finding container 1740ea01801f15d32990509c36ba32272ff25f243a53d08a190b299ddc413d83: Status 404 returned error can't find the container with id 1740ea01801f15d32990509c36ba32272ff25f243a53d08a190b299ddc413d83
Jul 13 21:42:04.318112 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:04.318074    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx" event={"ID":"10b802c1-0996-4c3a-92b1-03c96f602cb7","Type":"ContainerStarted","Data":"1740ea01801f15d32990509c36ba32272ff25f243a53d08a190b299ddc413d83"}
Jul 13 21:42:05.324873 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:05.324840    2564 generic.go:356] "Generic (PLEG): container finished" podID="10b802c1-0996-4c3a-92b1-03c96f602cb7" containerID="dca93dc09fb2538558a7f5ed02a38211fd3068fed183ab23e8eb67d054362399" exitCode=0
Jul 13 21:42:05.325255 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:05.324906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx" event={"ID":"10b802c1-0996-4c3a-92b1-03c96f602cb7","Type":"ContainerDied","Data":"dca93dc09fb2538558a7f5ed02a38211fd3068fed183ab23e8eb67d054362399"}
Jul 13 21:42:06.257455 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.257419    2564 patch_prober.go:28] interesting pod/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Jul 13 21:42:06.257678 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.257472    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Jul 13 21:42:06.330554 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.330525    2564 generic.go:356] "Generic (PLEG): container finished" podID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerID="507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18" exitCode=0
Jul 13 21:42:06.330913 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.330612    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" event={"ID":"d729c9da-a627-4131-8bf2-b839b3a1afdb","Type":"ContainerDied","Data":"507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18"}
Jul 13 21:42:06.331210 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.330996    2564 scope.go:117] "RemoveContainer" containerID="507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18"
Jul 13 21:42:06.332004 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.331975    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:42:06.332117 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.332048    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:42:06.332117 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.332079    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:42:06.471203 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.471178    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"
Jul 13 21:42:06.523383 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.523279    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b802c1-0996-4c3a-92b1-03c96f602cb7-maas-api-service-ca\") pod \"10b802c1-0996-4c3a-92b1-03c96f602cb7\" (UID: \"10b802c1-0996-4c3a-92b1-03c96f602cb7\") "
Jul 13 21:42:06.523616 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.523568    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b802c1-0996-4c3a-92b1-03c96f602cb7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10b802c1-0996-4c3a-92b1-03c96f602cb7" (UID: "10b802c1-0996-4c3a-92b1-03c96f602cb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:42:06.624676 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:06.624634    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b802c1-0996-4c3a-92b1-03c96f602cb7-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:07.337115 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:07.337071    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" event={"ID":"d729c9da-a627-4131-8bf2-b839b3a1afdb","Type":"ContainerStarted","Data":"c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747"}
Jul 13 21:42:07.338378 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:07.338353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx" event={"ID":"10b802c1-0996-4c3a-92b1-03c96f602cb7","Type":"ContainerDied","Data":"1740ea01801f15d32990509c36ba32272ff25f243a53d08a190b299ddc413d83"}
Jul 13 21:42:07.338504 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:07.338397    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1740ea01801f15d32990509c36ba32272ff25f243a53d08a190b299ddc413d83"
Jul 13 21:42:07.338504 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:07.338360    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx"
Jul 13 21:42:08.212336 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:08.212304    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:42:08.212541 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:08.212512    2564 patch_prober.go:28] interesting pod/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Jul 13 21:42:08.212606 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:08.212564    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Jul 13 21:42:09.211951 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:09.211914    2564 patch_prober.go:28] interesting pod/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Jul 13 21:42:09.212360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:09.211975    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Jul 13 21:42:10.211855 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:10.211818    2564 patch_prober.go:28] interesting pod/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Jul 13 21:42:10.212030 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:10.211886    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Jul 13 21:42:11.215561 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:11.215525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:42:11.215986 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:11.215791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:42:11.216382 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:11.216364    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:42:14.028404 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.028370    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"]
Jul 13 21:42:14.028904 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.028644    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp" podUID="8c0e0510-697e-4b90-b94a-cccdf6a30059" containerName="maas-api" containerID="cri-o://f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102" gracePeriod=30
Jul 13 21:42:14.125467 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.125438    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:14.266903 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.266879    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:42:14.381127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.381051    2564 generic.go:356] "Generic (PLEG): container finished" podID="8c0e0510-697e-4b90-b94a-cccdf6a30059" containerID="f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102" exitCode=0
Jul 13 21:42:14.381127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.381096    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"
Jul 13 21:42:14.381320 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.381140    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp" event={"ID":"8c0e0510-697e-4b90-b94a-cccdf6a30059","Type":"ContainerDied","Data":"f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102"}
Jul 13 21:42:14.381320 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.381229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp" event={"ID":"8c0e0510-697e-4b90-b94a-cccdf6a30059","Type":"ContainerDied","Data":"6f76134d603210edc663ee4e201c7bd7a968348367202f6df2960332a656ce32"}
Jul 13 21:42:14.381320 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.381259    2564 scope.go:117] "RemoveContainer" containerID="f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102"
Jul 13 21:42:14.389669 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.389644    2564 scope.go:117] "RemoveContainer" containerID="f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102"
Jul 13 21:42:14.389918 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:42:14.389888    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102\": container with ID starting with f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102 not found: ID does not exist" containerID="f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102"
Jul 13 21:42:14.390006 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.389914    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102"} err="failed to get container status \"f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102\": rpc error: code = NotFound desc = could not find container \"f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102\": container with ID starting with f2846f38d2e98611f05fe00c0ffe0e89d24b29ef5e7faf494cfa4726753b4102 not found: ID does not exist"
Jul 13 21:42:14.391370 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.391357    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0e0510-697e-4b90-b94a-cccdf6a30059-openshift-service-ca\") pod \"8c0e0510-697e-4b90-b94a-cccdf6a30059\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") "
Jul 13 21:42:14.391428 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.391384    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0510-697e-4b90-b94a-cccdf6a30059-maas-api-tls\") pod \"8c0e0510-697e-4b90-b94a-cccdf6a30059\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") "
Jul 13 21:42:14.391470 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.391427    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhmrw\" (UniqueName: \"kubernetes.io/projected/8c0e0510-697e-4b90-b94a-cccdf6a30059-kube-api-access-dhmrw\") pod \"8c0e0510-697e-4b90-b94a-cccdf6a30059\" (UID: \"8c0e0510-697e-4b90-b94a-cccdf6a30059\") "
Jul 13 21:42:14.391673 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.391655    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0e0510-697e-4b90-b94a-cccdf6a30059-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c0e0510-697e-4b90-b94a-cccdf6a30059" (UID: "8c0e0510-697e-4b90-b94a-cccdf6a30059"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:42:14.393374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.393353    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0e0510-697e-4b90-b94a-cccdf6a30059-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c0e0510-697e-4b90-b94a-cccdf6a30059" (UID: "8c0e0510-697e-4b90-b94a-cccdf6a30059"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:42:14.393594 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.393560    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0e0510-697e-4b90-b94a-cccdf6a30059-kube-api-access-dhmrw" (OuterVolumeSpecName: "kube-api-access-dhmrw") pod "8c0e0510-697e-4b90-b94a-cccdf6a30059" (UID: "8c0e0510-697e-4b90-b94a-cccdf6a30059"). InnerVolumeSpecName "kube-api-access-dhmrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:42:14.492712 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.492689    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0510-697e-4b90-b94a-cccdf6a30059-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:14.492712 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.492709    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhmrw\" (UniqueName: \"kubernetes.io/projected/8c0e0510-697e-4b90-b94a-cccdf6a30059-kube-api-access-dhmrw\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:14.492840 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.492718    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0e0510-697e-4b90-b94a-cccdf6a30059-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:14.702211 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.702187    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"]
Jul 13 21:42:14.706732 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:14.706702    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8603bf9d-7fcfb97d6d-59rgp"]
Jul 13 21:42:15.821153 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:15.821116    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c0e0510-697e-4b90-b94a-cccdf6a30059" path="/var/lib/kubelet/pods/8c0e0510-697e-4b90-b94a-cccdf6a30059/volumes"
Jul 13 21:42:18.736642 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:18.736605    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:24.574309 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:24.574267    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:24.674857 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:24.674824    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:29.379325 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:29.379291    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"]
Jul 13 21:42:29.379830 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:29.379542    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy" containerID="cri-o://c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747" gracePeriod=30
Jul 13 21:42:29.443753 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:29.443721    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:29.643399 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:29.643311    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:29.771318 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:29.771275    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:29.870789 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:29.870755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:30.025401 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:30.025378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:34.629111 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.629089    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:42:34.734463 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734390    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729c9da-a627-4131-8bf2-b839b3a1afdb-istiod-ca-cert\") pod \"d729c9da-a627-4131-8bf2-b839b3a1afdb\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") "
Jul 13 21:42:34.734463 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734425    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-data\") pod \"d729c9da-a627-4131-8bf2-b839b3a1afdb\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") "
Jul 13 21:42:34.734463 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734457    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-token\") pod \"d729c9da-a627-4131-8bf2-b839b3a1afdb\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") "
Jul 13 21:42:34.734746 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734479    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-certs\") pod \"d729c9da-a627-4131-8bf2-b839b3a1afdb\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") "
Jul 13 21:42:34.734746 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734505    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-credential-socket\") pod \"d729c9da-a627-4131-8bf2-b839b3a1afdb\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") "
Jul 13 21:42:34.734746 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734560    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-podinfo\") pod \"d729c9da-a627-4131-8bf2-b839b3a1afdb\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") "
Jul 13 21:42:34.734746 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734621    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-socket\") pod \"d729c9da-a627-4131-8bf2-b839b3a1afdb\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") "
Jul 13 21:42:34.734746 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734643    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-envoy\") pod \"d729c9da-a627-4131-8bf2-b839b3a1afdb\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") "
Jul 13 21:42:34.734746 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734675    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8scnl\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-kube-api-access-8scnl\") pod \"d729c9da-a627-4131-8bf2-b839b3a1afdb\" (UID: \"d729c9da-a627-4131-8bf2-b839b3a1afdb\") "
Jul 13 21:42:34.735046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734816    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d729c9da-a627-4131-8bf2-b839b3a1afdb" (UID: "d729c9da-a627-4131-8bf2-b839b3a1afdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:42:34.735046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734840    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d729c9da-a627-4131-8bf2-b839b3a1afdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d729c9da-a627-4131-8bf2-b839b3a1afdb" (UID: "d729c9da-a627-4131-8bf2-b839b3a1afdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:42:34.735046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734924    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d729c9da-a627-4131-8bf2-b839b3a1afdb" (UID: "d729c9da-a627-4131-8bf2-b839b3a1afdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:42:34.735046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734914    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d729c9da-a627-4131-8bf2-b839b3a1afdb" (UID: "d729c9da-a627-4131-8bf2-b839b3a1afdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:42:34.735046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.734998    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729c9da-a627-4131-8bf2-b839b3a1afdb-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:34.735046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.735016    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:34.735046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.735030    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:34.735046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.735042    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:34.735448 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.735288    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "d729c9da-a627-4131-8bf2-b839b3a1afdb" (UID: "d729c9da-a627-4131-8bf2-b839b3a1afdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:42:34.736830 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.736803    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "d729c9da-a627-4131-8bf2-b839b3a1afdb" (UID: "d729c9da-a627-4131-8bf2-b839b3a1afdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:42:34.736934 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.736848    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-kube-api-access-8scnl" (OuterVolumeSpecName: "kube-api-access-8scnl") pod "d729c9da-a627-4131-8bf2-b839b3a1afdb" (UID: "d729c9da-a627-4131-8bf2-b839b3a1afdb"). InnerVolumeSpecName "kube-api-access-8scnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:42:34.737353 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.737329    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d729c9da-a627-4131-8bf2-b839b3a1afdb" (UID: "d729c9da-a627-4131-8bf2-b839b3a1afdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:42:34.737431 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.737330    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d729c9da-a627-4131-8bf2-b839b3a1afdb" (UID: "d729c9da-a627-4131-8bf2-b839b3a1afdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:42:34.835643 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.835617    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:34.835643 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.835639    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:34.835643 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.835648    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:34.835861 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.835657    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729c9da-a627-4131-8bf2-b839b3a1afdb-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:34.835861 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:34.835665    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8scnl\" (UniqueName: \"kubernetes.io/projected/d729c9da-a627-4131-8bf2-b839b3a1afdb-kube-api-access-8scnl\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:42:35.477957 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.477925    2564 generic.go:356] "Generic (PLEG): container finished" podID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerID="c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747" exitCode=0
Jul 13 21:42:35.478140 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.477978    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"
Jul 13 21:42:35.478140 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.478003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" event={"ID":"d729c9da-a627-4131-8bf2-b839b3a1afdb","Type":"ContainerDied","Data":"c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747"}
Jul 13 21:42:35.478140 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.478045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs" event={"ID":"d729c9da-a627-4131-8bf2-b839b3a1afdb","Type":"ContainerDied","Data":"238073d48e2046f0ebc7d51479c457c435aff74e2b15b2ef7cea0b08ecaf2e85"}
Jul 13 21:42:35.478140 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.478066    2564 scope.go:117] "RemoveContainer" containerID="c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747"
Jul 13 21:42:35.487141 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.487124    2564 scope.go:117] "RemoveContainer" containerID="507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18"
Jul 13 21:42:35.495132 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.494976    2564 scope.go:117] "RemoveContainer" containerID="c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747"
Jul 13 21:42:35.495241 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:42:35.495222    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747\": container with ID starting with c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747 not found: ID does not exist" containerID="c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747"
Jul 13 21:42:35.495288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.495249    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747"} err="failed to get container status \"c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747\": rpc error: code = NotFound desc = could not find container \"c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747\": container with ID starting with c1458ccd48a366240dfe5bc96522a979780502236b300a01640d4da61a48c747 not found: ID does not exist"
Jul 13 21:42:35.495288 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.495265    2564 scope.go:117] "RemoveContainer" containerID="507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18"
Jul 13 21:42:35.495467 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:42:35.495449    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18\": container with ID starting with 507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18 not found: ID does not exist" containerID="507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18"
Jul 13 21:42:35.495511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.495470    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18"} err="failed to get container status \"507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18\": rpc error: code = NotFound desc = could not find container \"507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18\": container with ID starting with 507e09f30c3e7afcddf3b3d8acf23e41529d199989edfe2b7a7b37418f83ea18 not found: ID does not exist"
Jul 13 21:42:35.498383 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.498360    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"]
Jul 13 21:42:35.502196 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.502177    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8603bf9d-openshift-default-58b478ff8c-k5wfs"]
Jul 13 21:42:35.586892 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.586855    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:35.690405 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.690376    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:35.787510 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.787481    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:35.821107 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:35.821080    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" path="/var/lib/kubelet/pods/d729c9da-a627-4131-8bf2-b839b3a1afdb/volumes"
Jul 13 21:42:36.048858 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:36.048763    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:41.620313 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:41.620273    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:41.720380 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:41.720352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:46.315039 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:46.314990    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:46.416869 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:46.416830    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:50.664140 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.664102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:50.952381 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.952292    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"]
Jul 13 21:42:50.953005 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.952979    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b802c1-0996-4c3a-92b1-03c96f602cb7" containerName="revoke-keys"
Jul 13 21:42:50.953162 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.953151    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b802c1-0996-4c3a-92b1-03c96f602cb7" containerName="revoke-keys"
Jul 13 21:42:50.953251 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.953242    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy"
Jul 13 21:42:50.953323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.953314    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy"
Jul 13 21:42:50.953399 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.953390    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy"
Jul 13 21:42:50.953486 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.953475    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy"
Jul 13 21:42:50.953610 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.953574    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0e0510-697e-4b90-b94a-cccdf6a30059" containerName="maas-api"
Jul 13 21:42:50.953723 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.953713    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0e0510-697e-4b90-b94a-cccdf6a30059" containerName="maas-api"
Jul 13 21:42:50.953915 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.953904    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="10b802c1-0996-4c3a-92b1-03c96f602cb7" containerName="revoke-keys"
Jul 13 21:42:50.954019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.954000    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0e0510-697e-4b90-b94a-cccdf6a30059" containerName="maas-api"
Jul 13 21:42:50.954019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.954017    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy"
Jul 13 21:42:50.956145 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.956124    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:50.958472 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.958452    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6c7d8be7-openshift-default-dockercfg-lrb8w\""
Jul 13 21:42:50.968247 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.968221    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"]
Jul 13 21:42:50.981115 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:50.981089    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:51.048040 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.047995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tb47\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-kube-api-access-7tb47\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.048040 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.048038    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-socket\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.048264 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.048064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361755b8-c324-4064-b2a5-560da7c8dd9a-istiod-ca-cert\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.048264 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.048160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-podinfo\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.048264 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.048212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-envoy\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.048424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.048264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-certs\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.048424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.048289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-token\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.048424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.048326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-data\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.048424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.048357    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-credential-socket\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149470 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149448    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tb47\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-kube-api-access-7tb47\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149575 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-socket\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149575 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361755b8-c324-4064-b2a5-560da7c8dd9a-istiod-ca-cert\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149575 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149527    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-podinfo\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149575 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-envoy\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149792 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149608    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-certs\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149792 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-token\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149792 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-data\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149792 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149703    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-credential-socket\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.149989 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.149901    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-socket\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.150051 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.150011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-credential-socket\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.150148 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.150130    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-data\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.150199 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.150129    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-certs\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.150405 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.150387    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361755b8-c324-4064-b2a5-560da7c8dd9a-istiod-ca-cert\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.151997 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.151973    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-envoy\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.152347 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.152330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-podinfo\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.158712 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.158687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-token\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.158824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.158807    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tb47\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-kube-api-access-7tb47\") pod \"e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") " pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.266693 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.266668    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:51.399748 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.399680    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"]
Jul 13 21:42:51.401980 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:42:51.401955    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361755b8_c324_4064_b2a5_560da7c8dd9a.slice/crio-649bfadf8607b3ae7286c76e2bbdca3939a990c07fe6adca16afa8e92e2cd425 WatchSource:0}: Error finding container 649bfadf8607b3ae7286c76e2bbdca3939a990c07fe6adca16afa8e92e2cd425: Status 404 returned error can't find the container with id 649bfadf8607b3ae7286c76e2bbdca3939a990c07fe6adca16afa8e92e2cd425
Jul 13 21:42:51.404020 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.403991    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:42:51.404105 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.404057    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:42:51.404105 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.404087    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:42:51.555356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.555187    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" event={"ID":"361755b8-c324-4064-b2a5-560da7c8dd9a","Type":"ContainerStarted","Data":"8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f"}
Jul 13 21:42:51.555356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.555248    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" event={"ID":"361755b8-c324-4064-b2a5-560da7c8dd9a","Type":"ContainerStarted","Data":"649bfadf8607b3ae7286c76e2bbdca3939a990c07fe6adca16afa8e92e2cd425"}
Jul 13 21:42:51.556155 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.556121    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:51.574093 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:51.573832    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" podStartSLOduration=1.573816305 podStartE2EDuration="1.573816305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:42:51.572736444 +0000 UTC m=+2018.230393450" watchObservedRunningTime="2026-07-13 21:42:51.573816305 +0000 UTC m=+2018.231473325"
Jul 13 21:42:52.267045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:52.267005    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:52.271665 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:52.271642    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:52.560659 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:52.560555    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:52.561389 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:52.561368    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:42:53.523333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.523293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:53.591087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.591060    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"]
Jul 13 21:42:53.591471 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.591457    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d729c9da-a627-4131-8bf2-b839b3a1afdb" containerName="istio-proxy"
Jul 13 21:42:53.593364 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.593346    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:53.595882 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.595855    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6c7d8be7-serving-cert\""
Jul 13 21:42:53.605663 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.605611    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"]
Jul 13 21:42:53.617753 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.617722    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:53.666334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.666309    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd73808-0482-437b-a52a-e2a02be1d1e2-maas-api-tls\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:53.666430 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.666365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4dm4\" (UniqueName: \"kubernetes.io/projected/0dd73808-0482-437b-a52a-e2a02be1d1e2-kube-api-access-h4dm4\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:53.666550 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.666530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd73808-0482-437b-a52a-e2a02be1d1e2-openshift-service-ca\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:53.682577 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.682550    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:53.767630 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.767598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd73808-0482-437b-a52a-e2a02be1d1e2-maas-api-tls\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:53.767788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.767637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4dm4\" (UniqueName: \"kubernetes.io/projected/0dd73808-0482-437b-a52a-e2a02be1d1e2-kube-api-access-h4dm4\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:53.767788 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.767670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd73808-0482-437b-a52a-e2a02be1d1e2-openshift-service-ca\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:53.767788 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:42:53.767758    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-serving-cert: secret "maas-api-e2e-mt-6c7d8be7-serving-cert" not found
Jul 13 21:42:53.767920 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:42:53.767854    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dd73808-0482-437b-a52a-e2a02be1d1e2-maas-api-tls podName:0dd73808-0482-437b-a52a-e2a02be1d1e2 nodeName:}" failed. No retries permitted until 2026-07-13 21:42:54.267838049 +0000 UTC m=+2020.925495037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0dd73808-0482-437b-a52a-e2a02be1d1e2-maas-api-tls") pod "maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv" (UID: "0dd73808-0482-437b-a52a-e2a02be1d1e2") : secret "maas-api-e2e-mt-6c7d8be7-serving-cert" not found
Jul 13 21:42:53.768313 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.768295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd73808-0482-437b-a52a-e2a02be1d1e2-openshift-service-ca\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:53.776971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:53.776919    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4dm4\" (UniqueName: \"kubernetes.io/projected/0dd73808-0482-437b-a52a-e2a02be1d1e2-kube-api-access-h4dm4\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:54.163993 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:54.163901    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:54.266259 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:54.266227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:54.273129 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:54.273106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd73808-0482-437b-a52a-e2a02be1d1e2-maas-api-tls\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:54.275522 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:54.275506    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd73808-0482-437b-a52a-e2a02be1d1e2-maas-api-tls\") pod \"maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:54.503691 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:54.503662    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:54.640367 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:54.640346    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"]
Jul 13 21:42:54.642641 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:42:54.642611    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd73808_0482_437b_a52a_e2a02be1d1e2.slice/crio-9809538fbdd01b7b98237ce905e096a98a0a1bf5161509e5aa2312d8853cae53 WatchSource:0}: Error finding container 9809538fbdd01b7b98237ce905e096a98a0a1bf5161509e5aa2312d8853cae53: Status 404 returned error can't find the container with id 9809538fbdd01b7b98237ce905e096a98a0a1bf5161509e5aa2312d8853cae53
Jul 13 21:42:55.579689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:55.579653    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv" event={"ID":"0dd73808-0482-437b-a52a-e2a02be1d1e2","Type":"ContainerStarted","Data":"f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07"}
Jul 13 21:42:55.579891 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:55.579713    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv" event={"ID":"0dd73808-0482-437b-a52a-e2a02be1d1e2","Type":"ContainerStarted","Data":"9809538fbdd01b7b98237ce905e096a98a0a1bf5161509e5aa2312d8853cae53"}
Jul 13 21:42:55.579891 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:55.579752    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:42:55.596635 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:55.596563    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv" podStartSLOduration=2.26417598 podStartE2EDuration="2.596548011s" totalImagesPullingTime="332.372031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:42:55.594207951 +0000 UTC m=+2022.251864974" watchObservedRunningTime="2026-07-13 21:42:55.596548011 +0000 UTC m=+2022.254205018"
Jul 13 21:42:58.035803 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:42:58.035765    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:43:01.590227 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:01.590201    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:43:07.562559 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:07.562527    2564 patch_prober.go:28] interesting pod/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 13 21:43:07.562967 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:07.562610    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 13 21:43:07.639294 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:07.639264    2564 generic.go:356] "Generic (PLEG): container finished" podID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerID="8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f" exitCode=0
Jul 13 21:43:07.639425 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:07.639331    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" event={"ID":"361755b8-c324-4064-b2a5-560da7c8dd9a","Type":"ContainerDied","Data":"8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f"}
Jul 13 21:43:07.639749 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:07.639733    2564 scope.go:117] "RemoveContainer" containerID="8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f"
Jul 13 21:43:07.640647 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:07.640622    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:43:07.640748 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:07.640684    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:43:07.646771 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:07.641132    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:43:08.648532 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:08.648485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" event={"ID":"361755b8-c324-4064-b2a5-560da7c8dd9a","Type":"ContainerStarted","Data":"c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a"}
Jul 13 21:43:09.267689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:09.267655    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:43:09.267877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:09.267685    2564 patch_prober.go:28] interesting pod/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 13 21:43:09.267877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:09.267743    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 13 21:43:10.267501 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:10.267467    2564 patch_prober.go:28] interesting pod/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 13 21:43:10.268011 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:10.267682    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 13 21:43:11.267949 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:11.267914    2564 patch_prober.go:28] interesting pod/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 13 21:43:11.268438 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:11.267987    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 13 21:43:12.267485 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:12.267446    2564 patch_prober.go:28] interesting pod/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 13 21:43:12.267716 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:12.267663    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 13 21:43:13.270904 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:13.270876    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:43:13.271296 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:13.271161    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:43:13.271793 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:13.271773    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:43:14.003595 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:14.003563    2564 scope.go:117] "RemoveContainer" containerID="ccf45c3884aaffe09709077287c20bce0e3adc97891d347c04e559c5fb17a879"
Jul 13 21:43:14.011703 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:14.011674    2564 scope.go:117] "RemoveContainer" containerID="41101aa8b2ab365c37eb56f9a38362a290d8921427ca7aacd88b1961f28c380f"
Jul 13 21:43:19.392372 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:19.392332    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:43:46.627550 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:46.627455    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"]
Jul 13 21:43:46.629946 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:46.629927    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"
Jul 13 21:43:46.632302 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:46.632282    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:43:46.638772 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:46.638750    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"]
Jul 13 21:43:46.682200 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:46.682173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7cf9ff-3f9a-4a25-ac27-615356e24fba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn\" (UID: \"da7cf9ff-3f9a-4a25-ac27-615356e24fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"
Jul 13 21:43:46.783573 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:46.783547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7cf9ff-3f9a-4a25-ac27-615356e24fba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn\" (UID: \"da7cf9ff-3f9a-4a25-ac27-615356e24fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"
Jul 13 21:43:46.784103 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:46.784084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7cf9ff-3f9a-4a25-ac27-615356e24fba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn\" (UID: \"da7cf9ff-3f9a-4a25-ac27-615356e24fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"
Jul 13 21:43:46.941729 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:46.941662    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"
Jul 13 21:43:47.072952 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:47.072929    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"]
Jul 13 21:43:47.075077 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:43:47.075050    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7cf9ff_3f9a_4a25_ac27_615356e24fba.slice/crio-e95491a90161187918855a258549d1fd5f222f2f53eb3376fe1df36f10f4696f WatchSource:0}: Error finding container e95491a90161187918855a258549d1fd5f222f2f53eb3376fe1df36f10f4696f: Status 404 returned error can't find the container with id e95491a90161187918855a258549d1fd5f222f2f53eb3376fe1df36f10f4696f
Jul 13 21:43:47.838112 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:47.838072    2564 generic.go:356] "Generic (PLEG): container finished" podID="da7cf9ff-3f9a-4a25-ac27-615356e24fba" containerID="537306b701a321d8d868a8bdb1520c251110f864c9c63bf1fa18b01068c3b433" exitCode=0
Jul 13 21:43:47.838456 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:47.838136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn" event={"ID":"da7cf9ff-3f9a-4a25-ac27-615356e24fba","Type":"ContainerDied","Data":"537306b701a321d8d868a8bdb1520c251110f864c9c63bf1fa18b01068c3b433"}
Jul 13 21:43:47.838456 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:47.838231    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn" event={"ID":"da7cf9ff-3f9a-4a25-ac27-615356e24fba","Type":"ContainerStarted","Data":"e95491a90161187918855a258549d1fd5f222f2f53eb3376fe1df36f10f4696f"}
Jul 13 21:43:48.973070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:48.973050    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"
Jul 13 21:43:49.002079 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:49.002050    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7cf9ff-3f9a-4a25-ac27-615356e24fba-maas-api-service-ca\") pod \"da7cf9ff-3f9a-4a25-ac27-615356e24fba\" (UID: \"da7cf9ff-3f9a-4a25-ac27-615356e24fba\") "
Jul 13 21:43:49.002390 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:49.002366    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7cf9ff-3f9a-4a25-ac27-615356e24fba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da7cf9ff-3f9a-4a25-ac27-615356e24fba" (UID: "da7cf9ff-3f9a-4a25-ac27-615356e24fba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:43:49.102659 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:49.102607    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7cf9ff-3f9a-4a25-ac27-615356e24fba-maas-api-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:43:49.847727 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:49.847705    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn"
Jul 13 21:43:49.847727 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:49.847715    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn" event={"ID":"da7cf9ff-3f9a-4a25-ac27-615356e24fba","Type":"ContainerDied","Data":"e95491a90161187918855a258549d1fd5f222f2f53eb3376fe1df36f10f4696f"}
Jul 13 21:43:49.847906 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:49.847767    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e95491a90161187918855a258549d1fd5f222f2f53eb3376fe1df36f10f4696f"
Jul 13 21:43:57.021962 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.021928    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"]
Jul 13 21:43:57.022425 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.022194    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv" podUID="0dd73808-0482-437b-a52a-e2a02be1d1e2" containerName="maas-api" containerID="cri-o://f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07" gracePeriod=30
Jul 13 21:43:57.044545 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.044512    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:43:57.281615 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.281542    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:43:57.368265 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.368240    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd73808-0482-437b-a52a-e2a02be1d1e2-maas-api-tls\") pod \"0dd73808-0482-437b-a52a-e2a02be1d1e2\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") "
Jul 13 21:43:57.368390 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.368272    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4dm4\" (UniqueName: \"kubernetes.io/projected/0dd73808-0482-437b-a52a-e2a02be1d1e2-kube-api-access-h4dm4\") pod \"0dd73808-0482-437b-a52a-e2a02be1d1e2\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") "
Jul 13 21:43:57.368457 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.368403    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd73808-0482-437b-a52a-e2a02be1d1e2-openshift-service-ca\") pod \"0dd73808-0482-437b-a52a-e2a02be1d1e2\" (UID: \"0dd73808-0482-437b-a52a-e2a02be1d1e2\") "
Jul 13 21:43:57.368757 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.368730    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd73808-0482-437b-a52a-e2a02be1d1e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0dd73808-0482-437b-a52a-e2a02be1d1e2" (UID: "0dd73808-0482-437b-a52a-e2a02be1d1e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:43:57.370455 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.370431    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd73808-0482-437b-a52a-e2a02be1d1e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0dd73808-0482-437b-a52a-e2a02be1d1e2" (UID: "0dd73808-0482-437b-a52a-e2a02be1d1e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:43:57.370538 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.370465    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd73808-0482-437b-a52a-e2a02be1d1e2-kube-api-access-h4dm4" (OuterVolumeSpecName: "kube-api-access-h4dm4") pod "0dd73808-0482-437b-a52a-e2a02be1d1e2" (UID: "0dd73808-0482-437b-a52a-e2a02be1d1e2"). InnerVolumeSpecName "kube-api-access-h4dm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:43:57.469727 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.469705    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd73808-0482-437b-a52a-e2a02be1d1e2-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:43:57.469727 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.469725    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4dm4\" (UniqueName: \"kubernetes.io/projected/0dd73808-0482-437b-a52a-e2a02be1d1e2-kube-api-access-h4dm4\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:43:57.469844 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.469736    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd73808-0482-437b-a52a-e2a02be1d1e2-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:43:57.894091 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.894061    2564 generic.go:356] "Generic (PLEG): container finished" podID="0dd73808-0482-437b-a52a-e2a02be1d1e2" containerID="f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07" exitCode=0
Jul 13 21:43:57.894215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.894104    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"
Jul 13 21:43:57.894215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.894117    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv" event={"ID":"0dd73808-0482-437b-a52a-e2a02be1d1e2","Type":"ContainerDied","Data":"f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07"}
Jul 13 21:43:57.894215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.894159    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv" event={"ID":"0dd73808-0482-437b-a52a-e2a02be1d1e2","Type":"ContainerDied","Data":"9809538fbdd01b7b98237ce905e096a98a0a1bf5161509e5aa2312d8853cae53"}
Jul 13 21:43:57.894215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.894187    2564 scope.go:117] "RemoveContainer" containerID="f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07"
Jul 13 21:43:57.902764 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.902713    2564 scope.go:117] "RemoveContainer" containerID="f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07"
Jul 13 21:43:57.902988 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:43:57.902966    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07\": container with ID starting with f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07 not found: ID does not exist" containerID="f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07"
Jul 13 21:43:57.903029 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.902998    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07"} err="failed to get container status \"f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07\": rpc error: code = NotFound desc = could not find container \"f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07\": container with ID starting with f1e33c09d2abb6940f9bc65e643333a26a06c1db7202401828a17f031cf45c07 not found: ID does not exist"
Jul 13 21:43:57.909813 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.909791    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"]
Jul 13 21:43:57.913128 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:57.913106    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6c7d8be7-64c5cc8b64-vh5pv"]
Jul 13 21:43:59.822399 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:43:59.822361    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dd73808-0482-437b-a52a-e2a02be1d1e2" path="/var/lib/kubelet/pods/0dd73808-0482-437b-a52a-e2a02be1d1e2/volumes"
Jul 13 21:44:01.720165 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:01.720125    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:06.741458 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:06.741424    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:13.923088 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:13.923060    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:44:13.928915 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:13.928896    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:44:14.038862 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:14.038840    2564 scope.go:117] "RemoveContainer" containerID="e9566ffaba70e0d5dc5d79081eb93eb91962b6cd9cd715d5e31c2a094c0b68b8"
Jul 13 21:44:14.549385 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:14.549351    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:14.651787 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:14.651751    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:42.897473 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:42.897436    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:43.729461 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:43.729422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:49.363025 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:44:49.362991    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:00.136435 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.136400    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq"]
Jul 13 21:45:00.136827 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.136719    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7cf9ff-3f9a-4a25-ac27-615356e24fba" containerName="revoke-keys"
Jul 13 21:45:00.136827 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.136730    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7cf9ff-3f9a-4a25-ac27-615356e24fba" containerName="revoke-keys"
Jul 13 21:45:00.136827 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.136750    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd73808-0482-437b-a52a-e2a02be1d1e2" containerName="maas-api"
Jul 13 21:45:00.136827 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.136755    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd73808-0482-437b-a52a-e2a02be1d1e2" containerName="maas-api"
Jul 13 21:45:00.136827 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.136807    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7cf9ff-3f9a-4a25-ac27-615356e24fba" containerName="revoke-keys"
Jul 13 21:45:00.136827 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.136817    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd73808-0482-437b-a52a-e2a02be1d1e2" containerName="maas-api"
Jul 13 21:45:00.139667 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.139649    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq"
Jul 13 21:45:00.142242 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.142220    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xxvfz\""
Jul 13 21:45:00.147276 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.147253    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq"]
Jul 13 21:45:00.150100 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.150077    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq"
Jul 13 21:45:00.288663 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:00.288637    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq"]
Jul 13 21:45:00.290391 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:45:00.290363    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f68d2c_5bbf_4516_bab4_8b4781e055c3.slice/crio-3f58d0054fbb406ddd8c6ea538cbef5f6cf60c4874624561bcb6ff6b456d859c WatchSource:0}: Error finding container 3f58d0054fbb406ddd8c6ea538cbef5f6cf60c4874624561bcb6ff6b456d859c: Status 404 returned error can't find the container with id 3f58d0054fbb406ddd8c6ea538cbef5f6cf60c4874624561bcb6ff6b456d859c
Jul 13 21:45:01.183993 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:01.183954    2564 generic.go:356] "Generic (PLEG): container finished" podID="86f68d2c-5bbf-4516-bab4-8b4781e055c3" containerID="6f6a7f201fb88c875896f146211f13240f95e867d0dbcb55d44048a521142a39" exitCode=0
Jul 13 21:45:01.184286 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:01.184075    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq" event={"ID":"86f68d2c-5bbf-4516-bab4-8b4781e055c3","Type":"ContainerDied","Data":"6f6a7f201fb88c875896f146211f13240f95e867d0dbcb55d44048a521142a39"}
Jul 13 21:45:01.184286 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:01.184125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq" event={"ID":"86f68d2c-5bbf-4516-bab4-8b4781e055c3","Type":"ContainerStarted","Data":"3f58d0054fbb406ddd8c6ea538cbef5f6cf60c4874624561bcb6ff6b456d859c"}
Jul 13 21:45:02.318519 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:02.318495    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq"
Jul 13 21:45:03.194855 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:03.194826    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq"
Jul 13 21:45:03.195047 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:03.194862    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-qlkmq" event={"ID":"86f68d2c-5bbf-4516-bab4-8b4781e055c3","Type":"ContainerDied","Data":"3f58d0054fbb406ddd8c6ea538cbef5f6cf60c4874624561bcb6ff6b456d859c"}
Jul 13 21:45:03.195047 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:03.194913    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f58d0054fbb406ddd8c6ea538cbef5f6cf60c4874624561bcb6ff6b456d859c"
Jul 13 21:45:03.895019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:03.894985    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:04.002274 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:04.002242    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:13.867832 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:13.867753    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"]
Jul 13 21:45:13.868275 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:13.867996    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy" containerID="cri-o://c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a" gracePeriod=30
Jul 13 21:45:13.890893 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:13.890863    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:14.104554 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:14.104516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:15.129194 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:15.129144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:15.229456 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:15.229423    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:15.327598 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:15.327556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:15.996831 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:15.996798    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:16.147405 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:16.147377    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:16.317213 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:16.317122    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:17.511992 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:17.511957    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:19.117536 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.117516    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:45:19.167830 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.167762    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-token\") pod \"361755b8-c324-4064-b2a5-560da7c8dd9a\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") "
Jul 13 21:45:19.167830 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.167806    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-socket\") pod \"361755b8-c324-4064-b2a5-560da7c8dd9a\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") "
Jul 13 21:45:19.168020 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.167849    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-envoy\") pod \"361755b8-c324-4064-b2a5-560da7c8dd9a\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") "
Jul 13 21:45:19.168020 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.167880    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-podinfo\") pod \"361755b8-c324-4064-b2a5-560da7c8dd9a\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") "
Jul 13 21:45:19.168020 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.167932    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tb47\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-kube-api-access-7tb47\") pod \"361755b8-c324-4064-b2a5-560da7c8dd9a\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") "
Jul 13 21:45:19.168020 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.167955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361755b8-c324-4064-b2a5-560da7c8dd9a-istiod-ca-cert\") pod \"361755b8-c324-4064-b2a5-560da7c8dd9a\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") "
Jul 13 21:45:19.168020 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.167986    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-data\") pod \"361755b8-c324-4064-b2a5-560da7c8dd9a\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") "
Jul 13 21:45:19.168265 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.168024    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-certs\") pod \"361755b8-c324-4064-b2a5-560da7c8dd9a\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") "
Jul 13 21:45:19.168265 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.168049    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-credential-socket\") pod \"361755b8-c324-4064-b2a5-560da7c8dd9a\" (UID: \"361755b8-c324-4064-b2a5-560da7c8dd9a\") "
Jul 13 21:45:19.168374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.168246    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "361755b8-c324-4064-b2a5-560da7c8dd9a" (UID: "361755b8-c324-4064-b2a5-560da7c8dd9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:45:19.168523 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.168503    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "361755b8-c324-4064-b2a5-560da7c8dd9a" (UID: "361755b8-c324-4064-b2a5-560da7c8dd9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:45:19.168667 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.168648    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "361755b8-c324-4064-b2a5-560da7c8dd9a" (UID: "361755b8-c324-4064-b2a5-560da7c8dd9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:45:19.169470 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.169436    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "361755b8-c324-4064-b2a5-560da7c8dd9a" (UID: "361755b8-c324-4064-b2a5-560da7c8dd9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:45:19.169665 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.169633    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361755b8-c324-4064-b2a5-560da7c8dd9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "361755b8-c324-4064-b2a5-560da7c8dd9a" (UID: "361755b8-c324-4064-b2a5-560da7c8dd9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:45:19.170143 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.170110    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "361755b8-c324-4064-b2a5-560da7c8dd9a" (UID: "361755b8-c324-4064-b2a5-560da7c8dd9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:45:19.170374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.170351    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "361755b8-c324-4064-b2a5-560da7c8dd9a" (UID: "361755b8-c324-4064-b2a5-560da7c8dd9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:45:19.170510 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.170489    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-kube-api-access-7tb47" (OuterVolumeSpecName: "kube-api-access-7tb47") pod "361755b8-c324-4064-b2a5-560da7c8dd9a" (UID: "361755b8-c324-4064-b2a5-560da7c8dd9a"). InnerVolumeSpecName "kube-api-access-7tb47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:45:19.170684 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.170661    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "361755b8-c324-4064-b2a5-560da7c8dd9a" (UID: "361755b8-c324-4064-b2a5-560da7c8dd9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:45:19.268668 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.268648    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tb47\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-kube-api-access-7tb47\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:45:19.268776 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.268673    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361755b8-c324-4064-b2a5-560da7c8dd9a-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:45:19.268776 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.268687    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:45:19.268776 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.268699    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:45:19.268776 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.268711    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:45:19.268776 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.268719    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:45:19.268776 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.268727    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:45:19.268776 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.268738    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:45:19.268776 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.268748    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361755b8-c324-4064-b2a5-560da7c8dd9a-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:45:19.269832 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.269800    2564 generic.go:356] "Generic (PLEG): container finished" podID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerID="c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a" exitCode=0
Jul 13 21:45:19.269933 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.269836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" event={"ID":"361755b8-c324-4064-b2a5-560da7c8dd9a","Type":"ContainerDied","Data":"c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a"}
Jul 13 21:45:19.269933 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.269849    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"
Jul 13 21:45:19.269933 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.269896    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn" event={"ID":"361755b8-c324-4064-b2a5-560da7c8dd9a","Type":"ContainerDied","Data":"649bfadf8607b3ae7286c76e2bbdca3939a990c07fe6adca16afa8e92e2cd425"}
Jul 13 21:45:19.269933 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.269922    2564 scope.go:117] "RemoveContainer" containerID="c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a"
Jul 13 21:45:19.279452 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.279435    2564 scope.go:117] "RemoveContainer" containerID="8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f"
Jul 13 21:45:19.287641 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.287609    2564 scope.go:117] "RemoveContainer" containerID="c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a"
Jul 13 21:45:19.287905 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:45:19.287886    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a\": container with ID starting with c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a not found: ID does not exist" containerID="c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a"
Jul 13 21:45:19.287965 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.287913    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a"} err="failed to get container status \"c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a\": rpc error: code = NotFound desc = could not find container \"c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a\": container with ID starting with c2f12c285869a3d7732f672dda88140fc72cf9d6eee06652d8688e33fd1d729a not found: ID does not exist"
Jul 13 21:45:19.287965 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.287930    2564 scope.go:117] "RemoveContainer" containerID="8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f"
Jul 13 21:45:19.288167 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:45:19.288148    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f\": container with ID starting with 8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f not found: ID does not exist" containerID="8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f"
Jul 13 21:45:19.288216 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.288171    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f"} err="failed to get container status \"8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f\": rpc error: code = NotFound desc = could not find container \"8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f\": container with ID starting with 8e729fa274d0fc2391fa8b4393b9d38384f7c4643c80acea7662119d7b366f2f not found: ID does not exist"
Jul 13 21:45:19.293209 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.293187    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"]
Jul 13 21:45:19.298282 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.298262    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6c7d8be7-openshift-default-9746c6bf5-7bkpn"]
Jul 13 21:45:19.820939 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:19.820911    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" path="/var/lib/kubelet/pods/361755b8-c324-4064-b2a5-560da7c8dd9a/volumes"
Jul 13 21:45:23.394770 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:23.394731    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:23.461127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:23.461101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:24.071502 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:24.071455    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:30.062827 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:30.062790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:30.154729 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:30.154698    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:40.621339 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:40.621297    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:40.976325 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:45:40.976233    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:01.954217 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:01.954180    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:07.625360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:07.625321    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:07.684183 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:07.684154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:07.957559 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:07.957473    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:08.239677 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.239651    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"]
Jul 13 21:46:08.239958 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.239947    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy"
Jul 13 21:46:08.240005 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.239960    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy"
Jul 13 21:46:08.240005 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.239969    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f68d2c-5bbf-4516-bab4-8b4781e055c3" containerName="cleanup"
Jul 13 21:46:08.240005 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.239974    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f68d2c-5bbf-4516-bab4-8b4781e055c3" containerName="cleanup"
Jul 13 21:46:08.240005 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.239981    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy"
Jul 13 21:46:08.240005 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.239986    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy"
Jul 13 21:46:08.240170 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.240036    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy"
Jul 13 21:46:08.240170 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.240044    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f68d2c-5bbf-4516-bab4-8b4781e055c3" containerName="cleanup"
Jul 13 21:46:08.240170 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.240050    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="361755b8-c324-4064-b2a5-560da7c8dd9a" containerName="istio-proxy"
Jul 13 21:46:08.242949 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.242933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.245849 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.245829    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8c4d43-openshift-default-dockercfg-d85gc\""
Jul 13 21:46:08.263434 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.263415    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"]
Jul 13 21:46:08.279264 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.279232    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:08.333307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.333277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-socket\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.333415 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.333314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbgt\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-kube-api-access-2fbgt\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.333415 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.333347    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-certs\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.333511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.333418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-envoy\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.333511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.333447    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-data\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.333511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.333481    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-podinfo\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.333642 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.333579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-token\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.333706 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.333641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37cc970-b251-40ac-8b25-eb7f36a68b76-istiod-ca-cert\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.333706 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.333672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-credential-socket\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434191 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434166    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-token\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37cc970-b251-40ac-8b25-eb7f36a68b76-istiod-ca-cert\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-credential-socket\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-socket\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbgt\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-kube-api-access-2fbgt\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434310 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-certs\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434577 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-envoy\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434577 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434342    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-data\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434577 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434380    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-podinfo\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434808 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434783    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-data\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434874 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434804    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-credential-socket\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434921 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434903    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-socket\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.434962 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.434909    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-certs\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.435159 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.435138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37cc970-b251-40ac-8b25-eb7f36a68b76-istiod-ca-cert\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.436922 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.436890    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-envoy\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.437145 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.437124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-podinfo\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.444625 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.444604    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-token\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.444770 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.444647    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbgt\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-kube-api-access-2fbgt\") pod \"e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") " pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.506536 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.506460    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:08.553802 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.553762    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:08.693334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.693312    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"]
Jul 13 21:46:08.695394 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:46:08.695366    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37cc970_b251_40ac_8b25_eb7f36a68b76.slice/crio-02a8d3b11ffc62911062436c892bd6a97a9cc06c9c31d13a455575ff0ab15267 WatchSource:0}: Error finding container 02a8d3b11ffc62911062436c892bd6a97a9cc06c9c31d13a455575ff0ab15267: Status 404 returned error can't find the container with id 02a8d3b11ffc62911062436c892bd6a97a9cc06c9c31d13a455575ff0ab15267
Jul 13 21:46:08.697535 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.697496    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:08.697654 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.697599    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:08.697654 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:08.697630    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:09.510086 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:09.510052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" event={"ID":"c37cc970-b251-40ac-8b25-eb7f36a68b76","Type":"ContainerStarted","Data":"4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429"}
Jul 13 21:46:09.510299 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:09.510122    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" event={"ID":"c37cc970-b251-40ac-8b25-eb7f36a68b76","Type":"ContainerStarted","Data":"02a8d3b11ffc62911062436c892bd6a97a9cc06c9c31d13a455575ff0ab15267"}
Jul 13 21:46:09.528152 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:09.528110    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" podStartSLOduration=1.528098789 podStartE2EDuration="1.528098789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:46:09.527371697 +0000 UTC m=+2216.185028711" watchObservedRunningTime="2026-07-13 21:46:09.528098789 +0000 UTC m=+2216.185755795"
Jul 13 21:46:09.554639 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:09.554611    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:09.559330 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:09.559310    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:10.514953 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:10.514920    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:10.515961 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:10.515938    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:11.384218 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.384179    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:11.478795 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.478761    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:11.479674 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.479655    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"]
Jul 13 21:46:11.482920 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.482895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.485704 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.485683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8c4d43-serving-cert\""
Jul 13 21:46:11.492188 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.492167    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"]
Jul 13 21:46:11.545236 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.545191    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:11.660050 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.659967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.660050 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.660031    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.660272 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.660121    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qxcs\" (UniqueName: \"kubernetes.io/projected/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-kube-api-access-2qxcs\") pod \"maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.761114 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.761087    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qxcs\" (UniqueName: \"kubernetes.io/projected/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-kube-api-access-2qxcs\") pod \"maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.761258 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.761141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.761258 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.761168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.761773 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.761750    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.763777 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.763753    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.768429 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.768399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:11.769394 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.769372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qxcs\" (UniqueName: \"kubernetes.io/projected/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-kube-api-access-2qxcs\") pod \"maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.794235 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.794216    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:11.929483 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.929459    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"]
Jul 13 21:46:11.931539 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:46:11.931504    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8503e90_2c4d_4bd8_bb55_f8fe0403de69.slice/crio-46ecf7c43e4992da1cc1c01a29067442bcc8c762316124dae4208e3ad9172088 WatchSource:0}: Error finding container 46ecf7c43e4992da1cc1c01a29067442bcc8c762316124dae4208e3ad9172088: Status 404 returned error can't find the container with id 46ecf7c43e4992da1cc1c01a29067442bcc8c762316124dae4208e3ad9172088
Jul 13 21:46:11.955073 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:11.955015    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:12.053969 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:12.053937    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:12.528641 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:12.528613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9" event={"ID":"e8503e90-2c4d-4bd8-bb55-f8fe0403de69","Type":"ContainerStarted","Data":"46ecf7c43e4992da1cc1c01a29067442bcc8c762316124dae4208e3ad9172088"}
Jul 13 21:46:13.536774 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:13.536744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9" event={"ID":"e8503e90-2c4d-4bd8-bb55-f8fe0403de69","Type":"ContainerStarted","Data":"079e53cde652c251b701358ae07a05af362dd97edd1d14e8026446fb5825c02c"}
Jul 13 21:46:13.537189 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:13.536818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:13.556332 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:13.556279    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9" podStartSLOduration=2.006523614 podStartE2EDuration="2.556263839s" totalImagesPullingTime="549.740225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:46:13.554597873 +0000 UTC m=+2220.212254874" watchObservedRunningTime="2026-07-13 21:46:13.556263839 +0000 UTC m=+2220.213920872"
Jul 13 21:46:14.066274 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:14.066239    2564 scope.go:117] "RemoveContainer" containerID="c5c0b6586edef4fa8de1e5793d0e15cd308f48d6890c73908372792df7ad40bf"
Jul 13 21:46:15.854066 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:15.854028    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:16.165374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.165285    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:16.423956 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.423873    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"]
Jul 13 21:46:16.435544 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.435514    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.436966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.436938    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"]
Jul 13 21:46:16.438164 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.438139    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2522e6-openshift-default-dockercfg-xx2jb\""
Jul 13 21:46:16.447000 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.446967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:16.598142 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.598111    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-podinfo\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.598142 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.598144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-envoy\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.598347 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.598162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istiod-ca-cert\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.598347 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.598179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-data\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.598347 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.598250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-credential-socket\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.598347 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.598287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-socket\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.598347 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.598324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-certs\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.598511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.598428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-token\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.598511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.598455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7ft\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-kube-api-access-9v7ft\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.699598 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699512    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-credential-socket\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.699598 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-socket\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.699598 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-certs\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.699826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-token\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.699826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7ft\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-kube-api-access-9v7ft\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.699826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699705    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-podinfo\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.699826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-envoy\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.699826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istiod-ca-cert\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.699826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-data\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.700154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.699987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-credential-socket\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.700154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.700016    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-socket\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.700154 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.700102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-data\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.700320 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.700239    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-certs\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.700517 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.700495    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istiod-ca-cert\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.702101 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.702078    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-envoy\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.702328 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.702308    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-podinfo\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.707465 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.707449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-token\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.707902 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.707881    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7ft\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-kube-api-access-9v7ft\") pod \"e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") " pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.746734 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.746713    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:16.882761 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.882735    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"]
Jul 13 21:46:16.884542 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:46:16.884517    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb80a2cb_0a28_4949_b448_2c9bf4df36d9.slice/crio-4e7a31defcb145d19668df29c402d05d9362d0f74c6e362a83b3fb7153d68a3e WatchSource:0}: Error finding container 4e7a31defcb145d19668df29c402d05d9362d0f74c6e362a83b3fb7153d68a3e: Status 404 returned error can't find the container with id 4e7a31defcb145d19668df29c402d05d9362d0f74c6e362a83b3fb7153d68a3e
Jul 13 21:46:16.887278 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.887248    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:16.887359 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.887310    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:16.887359 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:16.887339    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:17.386039 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:17.386006    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:17.558812 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:17.558770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" event={"ID":"cb80a2cb-0a28-4949-b448-2c9bf4df36d9","Type":"ContainerStarted","Data":"ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f"}
Jul 13 21:46:17.558984 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:17.558831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" event={"ID":"cb80a2cb-0a28-4949-b448-2c9bf4df36d9","Type":"ContainerStarted","Data":"4e7a31defcb145d19668df29c402d05d9362d0f74c6e362a83b3fb7153d68a3e"}
Jul 13 21:46:17.578694 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:17.578577    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" podStartSLOduration=1.5785601809999998 podStartE2EDuration="1.578560181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:46:17.578456675 +0000 UTC m=+2224.236113686" watchObservedRunningTime="2026-07-13 21:46:17.578560181 +0000 UTC m=+2224.236217188"
Jul 13 21:46:17.747529 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:17.747501    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:17.752299 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:17.752274    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:18.563688 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:18.563657    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:18.564636 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:18.564619    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:19.546480 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.546452    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:46:19.557636 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.557576    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:19.796197 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.796159    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"]
Jul 13 21:46:19.799608 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.799523    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:19.801874 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.801849    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2522e6-serving-cert\""
Jul 13 21:46:19.810430 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.810408    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"]
Jul 13 21:46:19.826347 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.826321    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63344789-0faa-4c9c-94aa-4bec46b5d224-maas-api-tls\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:19.826447 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.826358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63344789-0faa-4c9c-94aa-4bec46b5d224-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:19.826447 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.826379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wd29\" (UniqueName: \"kubernetes.io/projected/63344789-0faa-4c9c-94aa-4bec46b5d224-kube-api-access-4wd29\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:19.927922 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.927889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63344789-0faa-4c9c-94aa-4bec46b5d224-maas-api-tls\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:19.928101 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.927977    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63344789-0faa-4c9c-94aa-4bec46b5d224-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:19.928101 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.928013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wd29\" (UniqueName: \"kubernetes.io/projected/63344789-0faa-4c9c-94aa-4bec46b5d224-kube-api-access-4wd29\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:19.928101 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:46:19.928035    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-serving-cert: secret "maas-api-e2e-inf-b-2522e6-serving-cert" not found
Jul 13 21:46:19.928101 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:46:19.928098    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63344789-0faa-4c9c-94aa-4bec46b5d224-maas-api-tls podName:63344789-0faa-4c9c-94aa-4bec46b5d224 nodeName:}" failed. No retries permitted until 2026-07-13 21:46:20.428078953 +0000 UTC m=+2227.085735957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63344789-0faa-4c9c-94aa-4bec46b5d224-maas-api-tls") pod "maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw" (UID: "63344789-0faa-4c9c-94aa-4bec46b5d224") : secret "maas-api-e2e-inf-b-2522e6-serving-cert" not found
Jul 13 21:46:19.928675 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.928653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63344789-0faa-4c9c-94aa-4bec46b5d224-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:19.938980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:19.938945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wd29\" (UniqueName: \"kubernetes.io/projected/63344789-0faa-4c9c-94aa-4bec46b5d224-kube-api-access-4wd29\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:20.431901 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:20.431865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63344789-0faa-4c9c-94aa-4bec46b5d224-maas-api-tls\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:20.434508 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:20.434486    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63344789-0faa-4c9c-94aa-4bec46b5d224-maas-api-tls\") pod \"maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:20.713736 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:20.713515    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:20.892063 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:20.892031    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"]
Jul 13 21:46:20.893068 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:46:20.893041    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63344789_0faa_4c9c_94aa_4bec46b5d224.slice/crio-4fd6d6ba87e8fdc7c75d4b8e0f3b40492d824ac51d85634fd6225cd1cc2e104e WatchSource:0}: Error finding container 4fd6d6ba87e8fdc7c75d4b8e0f3b40492d824ac51d85634fd6225cd1cc2e104e: Status 404 returned error can't find the container with id 4fd6d6ba87e8fdc7c75d4b8e0f3b40492d824ac51d85634fd6225cd1cc2e104e
Jul 13 21:46:20.972783 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:20.972690    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:21.584336 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:21.584245    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw" event={"ID":"63344789-0faa-4c9c-94aa-4bec46b5d224","Type":"ContainerStarted","Data":"7919f31af59653d1ccc1453e63e0e3904c40b5a07aa7c34c493bc41cca8ca0ed"}
Jul 13 21:46:21.584336 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:21.584308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw" event={"ID":"63344789-0faa-4c9c-94aa-4bec46b5d224","Type":"ContainerStarted","Data":"4fd6d6ba87e8fdc7c75d4b8e0f3b40492d824ac51d85634fd6225cd1cc2e104e"}
Jul 13 21:46:21.584504 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:21.584349    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:21.602463 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:21.602410    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw" podStartSLOduration=2.220441501 podStartE2EDuration="2.60239456s" totalImagesPullingTime="381.953059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:46:21.600270944 +0000 UTC m=+2228.257927947" watchObservedRunningTime="2026-07-13 21:46:21.60239456 +0000 UTC m=+2228.260051601"
Jul 13 21:46:23.935173 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:23.935140    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:26.555738 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:26.555697    2564 patch_prober.go:28] interesting pod/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 13 21:46:26.556110 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:26.555771    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 13 21:46:27.595273 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:27.595192    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:46:27.617020 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:27.616990    2564 generic.go:356] "Generic (PLEG): container finished" podID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerID="4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429" exitCode=0
Jul 13 21:46:27.617158 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:27.617063    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" event={"ID":"c37cc970-b251-40ac-8b25-eb7f36a68b76","Type":"ContainerDied","Data":"4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429"}
Jul 13 21:46:27.617387 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:27.617373    2564 scope.go:117] "RemoveContainer" containerID="4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429"
Jul 13 21:46:27.618540 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:27.618507    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:27.618654 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:27.618608    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:27.618710 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:27.618650    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:28.623601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:28.623541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" event={"ID":"c37cc970-b251-40ac-8b25-eb7f36a68b76","Type":"ContainerStarted","Data":"2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1"}
Jul 13 21:46:29.554315 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:29.554276    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:29.554565 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:29.554397    2564 patch_prober.go:28] interesting pod/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 13 21:46:29.554565 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:29.554443    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 13 21:46:30.555057 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:30.555017    2564 patch_prober.go:28] interesting pod/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 13 21:46:30.555781 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:30.555746    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 13 21:46:31.554848 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:31.554327    2564 patch_prober.go:28] interesting pod/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 13 21:46:31.554848 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:31.554393    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 13 21:46:32.554457 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:32.554421    2564 patch_prober.go:28] interesting pod/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 13 21:46:32.555433 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:32.555042    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 13 21:46:33.558841 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:33.558813    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:33.559259 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:33.559129    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:33.559783 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:33.559764    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:46:33.935278 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:33.935189    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:34.032016 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:34.031979    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:34.394975 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:34.394763    2564 patch_prober.go:28] interesting pod/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:41092->10.132.0.53:15021: read: connection reset by peer" start-of-body=
Jul 13 21:46:34.394975 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:34.394824    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:41092->10.132.0.53:15021: read: connection reset by peer"
Jul 13 21:46:34.659305 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:34.659203    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerID="ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f" exitCode=0
Jul 13 21:46:34.659305 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:34.659282    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" event={"ID":"cb80a2cb-0a28-4949-b448-2c9bf4df36d9","Type":"ContainerDied","Data":"ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f"}
Jul 13 21:46:34.659887 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:34.659870    2564 scope.go:117] "RemoveContainer" containerID="ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f"
Jul 13 21:46:34.660859 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:34.660835    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:34.660922 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:34.660907    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:34.660969 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:34.660940    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:46:35.669504 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:35.669464    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" event={"ID":"cb80a2cb-0a28-4949-b448-2c9bf4df36d9","Type":"ContainerStarted","Data":"459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec"}
Jul 13 21:46:35.748110 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:35.748076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:35.748401 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:35.748117    2564 patch_prober.go:28] interesting pod/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 13 21:46:35.748612 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:35.748556    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 13 21:46:36.288896 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:36.288856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:36.747253 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:36.747219    2564 patch_prober.go:28] interesting pod/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 13 21:46:36.747834 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:36.747800    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 13 21:46:37.747969 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:37.747936    2564 patch_prober.go:28] interesting pod/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 13 21:46:37.748559 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:37.748527    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 13 21:46:38.747275 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:38.747239    2564 patch_prober.go:28] interesting pod/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 13 21:46:38.747614 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:38.747564    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 13 21:46:39.751238 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:39.751209    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:39.751743 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:39.751609    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:39.752295 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:39.752274    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:46:42.251065 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:42.251026    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:42.351653 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:46:42.351617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:07.000200 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:07.000164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:32.360167 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:32.360133    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:33.267195 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:33.267158    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:33.991405 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:33.991348    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:34.094841 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:34.094804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:40.690434 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:40.690391    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:43.759726 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:43.759693    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"]
Jul 13 21:47:43.760200 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:43.759947    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9" podUID="e8503e90-2c4d-4bd8-bb55-f8fe0403de69" containerName="maas-api" containerID="cri-o://079e53cde652c251b701358ae07a05af362dd97edd1d14e8026446fb5825c02c" gracePeriod=30
Jul 13 21:47:43.783872 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:43.783823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:44.000768 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.000727    2564 generic.go:356] "Generic (PLEG): container finished" podID="e8503e90-2c4d-4bd8-bb55-f8fe0403de69" containerID="079e53cde652c251b701358ae07a05af362dd97edd1d14e8026446fb5825c02c" exitCode=0
Jul 13 21:47:44.000852 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.000807    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9" event={"ID":"e8503e90-2c4d-4bd8-bb55-f8fe0403de69","Type":"ContainerDied","Data":"079e53cde652c251b701358ae07a05af362dd97edd1d14e8026446fb5825c02c"}
Jul 13 21:47:44.019266 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.019215    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:47:44.117254 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.117230    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qxcs\" (UniqueName: \"kubernetes.io/projected/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-kube-api-access-2qxcs\") pod \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") "
Jul 13 21:47:44.117376 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.117266    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-maas-api-tls\") pod \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") "
Jul 13 21:47:44.117376 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.117309    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-openshift-service-ca\") pod \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\" (UID: \"e8503e90-2c4d-4bd8-bb55-f8fe0403de69\") "
Jul 13 21:47:44.117746 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.117721    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8503e90-2c4d-4bd8-bb55-f8fe0403de69" (UID: "e8503e90-2c4d-4bd8-bb55-f8fe0403de69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:47:44.119410 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.119386    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8503e90-2c4d-4bd8-bb55-f8fe0403de69" (UID: "e8503e90-2c4d-4bd8-bb55-f8fe0403de69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:47:44.119516 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.119452    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-kube-api-access-2qxcs" (OuterVolumeSpecName: "kube-api-access-2qxcs") pod "e8503e90-2c4d-4bd8-bb55-f8fe0403de69" (UID: "e8503e90-2c4d-4bd8-bb55-f8fe0403de69"). InnerVolumeSpecName "kube-api-access-2qxcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:47:44.218186 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.218152    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:47:44.218186 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.218185    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qxcs\" (UniqueName: \"kubernetes.io/projected/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-kube-api-access-2qxcs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:47:44.218186 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:44.218195    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8503e90-2c4d-4bd8-bb55-f8fe0403de69-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:47:45.007374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:45.007344    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"
Jul 13 21:47:45.007866 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:45.007390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9" event={"ID":"e8503e90-2c4d-4bd8-bb55-f8fe0403de69","Type":"ContainerDied","Data":"46ecf7c43e4992da1cc1c01a29067442bcc8c762316124dae4208e3ad9172088"}
Jul 13 21:47:45.007866 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:45.007457    2564 scope.go:117] "RemoveContainer" containerID="079e53cde652c251b701358ae07a05af362dd97edd1d14e8026446fb5825c02c"
Jul 13 21:47:45.028529 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:45.028508    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"]
Jul 13 21:47:45.032404 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:45.032378    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c4d43-74846cd566-csqb9"]
Jul 13 21:47:45.821727 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:45.821694    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8503e90-2c4d-4bd8-bb55-f8fe0403de69" path="/var/lib/kubelet/pods/e8503e90-2c4d-4bd8-bb55-f8fe0403de69/volumes"
Jul 13 21:47:47.020213 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:47.020177    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:47.120208 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:47.120174    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:48.441491 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:48.441453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:54.295325 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:54.295290    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:54.396477 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:54.396439    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:59.090828 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:59.090796    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"]
Jul 13 21:47:59.091229 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:59.091011    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy" containerID="cri-o://2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1" gracePeriod=30
Jul 13 21:47:59.109015 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:59.108948    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:59.322749 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:47:59.322716    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:00.029950 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:00.029901    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:00.129604 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:00.129557    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:03.562181 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:03.562144    2564 patch_prober.go:28] interesting pod/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 21:48:03.562578 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:03.562228    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:48:04.343677 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.343645    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:48:04.475176 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475110    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-data\") pod \"c37cc970-b251-40ac-8b25-eb7f36a68b76\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") "
Jul 13 21:48:04.475176 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475145    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-certs\") pod \"c37cc970-b251-40ac-8b25-eb7f36a68b76\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") "
Jul 13 21:48:04.475375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475191    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-envoy\") pod \"c37cc970-b251-40ac-8b25-eb7f36a68b76\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") "
Jul 13 21:48:04.475375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475216    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37cc970-b251-40ac-8b25-eb7f36a68b76-istiod-ca-cert\") pod \"c37cc970-b251-40ac-8b25-eb7f36a68b76\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") "
Jul 13 21:48:04.475375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475238    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-credential-socket\") pod \"c37cc970-b251-40ac-8b25-eb7f36a68b76\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") "
Jul 13 21:48:04.475375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475266    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-socket\") pod \"c37cc970-b251-40ac-8b25-eb7f36a68b76\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") "
Jul 13 21:48:04.475375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475291    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fbgt\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-kube-api-access-2fbgt\") pod \"c37cc970-b251-40ac-8b25-eb7f36a68b76\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") "
Jul 13 21:48:04.475375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475317    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-podinfo\") pod \"c37cc970-b251-40ac-8b25-eb7f36a68b76\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") "
Jul 13 21:48:04.475375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475367    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-token\") pod \"c37cc970-b251-40ac-8b25-eb7f36a68b76\" (UID: \"c37cc970-b251-40ac-8b25-eb7f36a68b76\") "
Jul 13 21:48:04.475758 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475563    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c37cc970-b251-40ac-8b25-eb7f36a68b76" (UID: "c37cc970-b251-40ac-8b25-eb7f36a68b76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:04.475758 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475579    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c37cc970-b251-40ac-8b25-eb7f36a68b76" (UID: "c37cc970-b251-40ac-8b25-eb7f36a68b76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:04.475758 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475669    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37cc970-b251-40ac-8b25-eb7f36a68b76-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c37cc970-b251-40ac-8b25-eb7f36a68b76" (UID: "c37cc970-b251-40ac-8b25-eb7f36a68b76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:48:04.475917 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.475890    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c37cc970-b251-40ac-8b25-eb7f36a68b76" (UID: "c37cc970-b251-40ac-8b25-eb7f36a68b76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:04.476139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.476117    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-data" (OuterVolumeSpecName: "istio-data") pod "c37cc970-b251-40ac-8b25-eb7f36a68b76" (UID: "c37cc970-b251-40ac-8b25-eb7f36a68b76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:04.477790 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.477766    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-kube-api-access-2fbgt" (OuterVolumeSpecName: "kube-api-access-2fbgt") pod "c37cc970-b251-40ac-8b25-eb7f36a68b76" (UID: "c37cc970-b251-40ac-8b25-eb7f36a68b76"). InnerVolumeSpecName "kube-api-access-2fbgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:48:04.477892 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.477788    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-token" (OuterVolumeSpecName: "istio-token") pod "c37cc970-b251-40ac-8b25-eb7f36a68b76" (UID: "c37cc970-b251-40ac-8b25-eb7f36a68b76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:48:04.478047 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.478031    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c37cc970-b251-40ac-8b25-eb7f36a68b76" (UID: "c37cc970-b251-40ac-8b25-eb7f36a68b76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:04.478120 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.478106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c37cc970-b251-40ac-8b25-eb7f36a68b76" (UID: "c37cc970-b251-40ac-8b25-eb7f36a68b76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:48:04.576373 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.576347    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:04.576373 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.576368    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:04.576373 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.576376    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:04.576820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.576383    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:04.576820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.576391    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37cc970-b251-40ac-8b25-eb7f36a68b76-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:04.576820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.576399    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:04.576820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.576406    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37cc970-b251-40ac-8b25-eb7f36a68b76-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:04.576820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.576414    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fbgt\" (UniqueName: \"kubernetes.io/projected/c37cc970-b251-40ac-8b25-eb7f36a68b76-kube-api-access-2fbgt\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:04.576820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:04.576424    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37cc970-b251-40ac-8b25-eb7f36a68b76-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:05.106886 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.106797    2564 generic.go:356] "Generic (PLEG): container finished" podID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerID="2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1" exitCode=0
Jul 13 21:48:05.106886 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.106848    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"
Jul 13 21:48:05.107084 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.106884    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" event={"ID":"c37cc970-b251-40ac-8b25-eb7f36a68b76","Type":"ContainerDied","Data":"2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1"}
Jul 13 21:48:05.107084 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.106955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm" event={"ID":"c37cc970-b251-40ac-8b25-eb7f36a68b76","Type":"ContainerDied","Data":"02a8d3b11ffc62911062436c892bd6a97a9cc06c9c31d13a455575ff0ab15267"}
Jul 13 21:48:05.107084 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.106986    2564 scope.go:117] "RemoveContainer" containerID="2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1"
Jul 13 21:48:05.116419 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.116400    2564 scope.go:117] "RemoveContainer" containerID="4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429"
Jul 13 21:48:05.124500 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.124479    2564 scope.go:117] "RemoveContainer" containerID="2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1"
Jul 13 21:48:05.124958 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:48:05.124874    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1\": container with ID starting with 2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1 not found: ID does not exist" containerID="2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1"
Jul 13 21:48:05.124958 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.124912    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1"} err="failed to get container status \"2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1\": rpc error: code = NotFound desc = could not find container \"2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1\": container with ID starting with 2b6c2b41bbe14b92b278faf75553e7e4d53554b177f7f3150da361773325fdd1 not found: ID does not exist"
Jul 13 21:48:05.124958 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.124935    2564 scope.go:117] "RemoveContainer" containerID="4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429"
Jul 13 21:48:05.125350 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:48:05.125279    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429\": container with ID starting with 4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429 not found: ID does not exist" containerID="4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429"
Jul 13 21:48:05.125350 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.125310    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429"} err="failed to get container status \"4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429\": rpc error: code = NotFound desc = could not find container \"4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429\": container with ID starting with 4de8a4477d5202f02fae361dd53ba7ce9eac47afd573053f986c3df38841f429 not found: ID does not exist"
Jul 13 21:48:05.127355 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.127334    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"]
Jul 13 21:48:05.135598 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.131994    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8c4d43-openshift-default-654f95cc6f-brlsm"]
Jul 13 21:48:05.821789 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:05.821744    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" path="/var/lib/kubelet/pods/c37cc970-b251-40ac-8b25-eb7f36a68b76/volumes"
Jul 13 21:48:09.920424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:09.920391    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"]
Jul 13 21:48:09.920919 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:09.920598    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw" podUID="63344789-0faa-4c9c-94aa-4bec46b5d224" containerName="maas-api" containerID="cri-o://7919f31af59653d1ccc1453e63e0e3904c40b5a07aa7c34c493bc41cca8ca0ed" gracePeriod=30
Jul 13 21:48:09.947177 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:09.947151    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:10.133545 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.133515    2564 generic.go:356] "Generic (PLEG): container finished" podID="63344789-0faa-4c9c-94aa-4bec46b5d224" containerID="7919f31af59653d1ccc1453e63e0e3904c40b5a07aa7c34c493bc41cca8ca0ed" exitCode=0
Jul 13 21:48:10.133708 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.133606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw" event={"ID":"63344789-0faa-4c9c-94aa-4bec46b5d224","Type":"ContainerDied","Data":"7919f31af59653d1ccc1453e63e0e3904c40b5a07aa7c34c493bc41cca8ca0ed"}
Jul 13 21:48:10.168884 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.168862    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:48:10.318668 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.318646    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63344789-0faa-4c9c-94aa-4bec46b5d224-maas-api-tls\") pod \"63344789-0faa-4c9c-94aa-4bec46b5d224\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") "
Jul 13 21:48:10.318811 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.318676    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63344789-0faa-4c9c-94aa-4bec46b5d224-openshift-service-ca\") pod \"63344789-0faa-4c9c-94aa-4bec46b5d224\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") "
Jul 13 21:48:10.318811 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.318705    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wd29\" (UniqueName: \"kubernetes.io/projected/63344789-0faa-4c9c-94aa-4bec46b5d224-kube-api-access-4wd29\") pod \"63344789-0faa-4c9c-94aa-4bec46b5d224\" (UID: \"63344789-0faa-4c9c-94aa-4bec46b5d224\") "
Jul 13 21:48:10.318991 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.318968    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63344789-0faa-4c9c-94aa-4bec46b5d224-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63344789-0faa-4c9c-94aa-4bec46b5d224" (UID: "63344789-0faa-4c9c-94aa-4bec46b5d224"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:48:10.320812 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.320791    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63344789-0faa-4c9c-94aa-4bec46b5d224-kube-api-access-4wd29" (OuterVolumeSpecName: "kube-api-access-4wd29") pod "63344789-0faa-4c9c-94aa-4bec46b5d224" (UID: "63344789-0faa-4c9c-94aa-4bec46b5d224"). InnerVolumeSpecName "kube-api-access-4wd29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:48:10.320902 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.320857    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63344789-0faa-4c9c-94aa-4bec46b5d224-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63344789-0faa-4c9c-94aa-4bec46b5d224" (UID: "63344789-0faa-4c9c-94aa-4bec46b5d224"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:48:10.419989 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.419963    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63344789-0faa-4c9c-94aa-4bec46b5d224-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:10.419989 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.419986    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63344789-0faa-4c9c-94aa-4bec46b5d224-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:10.420111 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:10.419996    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wd29\" (UniqueName: \"kubernetes.io/projected/63344789-0faa-4c9c-94aa-4bec46b5d224-kube-api-access-4wd29\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:11.139865 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:11.139831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw" event={"ID":"63344789-0faa-4c9c-94aa-4bec46b5d224","Type":"ContainerDied","Data":"4fd6d6ba87e8fdc7c75d4b8e0f3b40492d824ac51d85634fd6225cd1cc2e104e"}
Jul 13 21:48:11.140301 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:11.139889    2564 scope.go:117] "RemoveContainer" containerID="7919f31af59653d1ccc1453e63e0e3904c40b5a07aa7c34c493bc41cca8ca0ed"
Jul 13 21:48:11.140301 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:11.139847    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"
Jul 13 21:48:11.160182 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:11.160154    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"]
Jul 13 21:48:11.164151 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:11.164127    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2522e6-55b4d48db7-4n4gw"]
Jul 13 21:48:11.821394 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:11.821357    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63344789-0faa-4c9c-94aa-4bec46b5d224" path="/var/lib/kubelet/pods/63344789-0faa-4c9c-94aa-4bec46b5d224/volumes"
Jul 13 21:48:14.507315 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:14.507273    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:20.306855 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:20.306818    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:20.365878 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:20.365850    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:25.133756 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.133717    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"]
Jul 13 21:48:25.134214 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.133970    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy" containerID="cri-o://459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec" gracePeriod=30
Jul 13 21:48:25.152856 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.152827    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:25.360327 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.360291    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:25.689238 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.689205    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:25.957313 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957230    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"]
Jul 13 21:48:25.957778 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957735    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy"
Jul 13 21:48:25.957778 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957758    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy"
Jul 13 21:48:25.957778 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957779    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63344789-0faa-4c9c-94aa-4bec46b5d224" containerName="maas-api"
Jul 13 21:48:25.958027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957787    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="63344789-0faa-4c9c-94aa-4bec46b5d224" containerName="maas-api"
Jul 13 21:48:25.958027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957805    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy"
Jul 13 21:48:25.958027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957814    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy"
Jul 13 21:48:25.958027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957827    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8503e90-2c4d-4bd8-bb55-f8fe0403de69" containerName="maas-api"
Jul 13 21:48:25.958027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957835    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8503e90-2c4d-4bd8-bb55-f8fe0403de69" containerName="maas-api"
Jul 13 21:48:25.958027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957923    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy"
Jul 13 21:48:25.958027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957935    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="63344789-0faa-4c9c-94aa-4bec46b5d224" containerName="maas-api"
Jul 13 21:48:25.958027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.957945    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8503e90-2c4d-4bd8-bb55-f8fe0403de69" containerName="maas-api"
Jul 13 21:48:25.960995 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.960963    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:25.963351 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.963329    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-509096-openshift-default-dockercfg-tp5n6\""
Jul 13 21:48:25.974408 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.974386    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"]
Jul 13 21:48:25.981271 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:25.981245    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:26.138845 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.138810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-token\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.139233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.138849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-data\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.139233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.138877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-envoy\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.139233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.138953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-certs\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.139233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.138983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45pc5\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-kube-api-access-45pc5\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.139233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.139023    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b889198-e09b-4030-8c01-7a4565bbb81d-istiod-ca-cert\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.139233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.139044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-socket\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.139233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.139076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-credential-socket\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.139233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.139134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-podinfo\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.213940 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.213829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:26.240198 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-token\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240364 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-data\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240364 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-envoy\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240364 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-certs\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240536 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45pc5\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-kube-api-access-45pc5\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240536 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240504    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b889198-e09b-4030-8c01-7a4565bbb81d-istiod-ca-cert\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240681 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-socket\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240681 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-credential-socket\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240681 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-podinfo\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240681 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240640    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-certs\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240884 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-data\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240943 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-socket\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.240999 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.240951    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-credential-socket\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.241447 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.241414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b889198-e09b-4030-8c01-7a4565bbb81d-istiod-ca-cert\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.242944 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.242918    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-envoy\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.243233 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.243216    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-podinfo\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.248277 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.248246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45pc5\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-kube-api-access-45pc5\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.248406 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.248387    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-token\") pod \"e2e-api-a-509096-openshift-default-59d6fff597-sp5xg\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") " pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.270505 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.270482    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:26.405257 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.405233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"]
Jul 13 21:48:26.406858 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:48:26.406829    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b889198_e09b_4030_8c01_7a4565bbb81d.slice/crio-fbebf4a98edaeace10d43ea0412103dcee765485a70ff69985406c7cd97af0f1 WatchSource:0}: Error finding container fbebf4a98edaeace10d43ea0412103dcee765485a70ff69985406c7cd97af0f1: Status 404 returned error can't find the container with id fbebf4a98edaeace10d43ea0412103dcee765485a70ff69985406c7cd97af0f1
Jul 13 21:48:26.408862 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.408831    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:26.408930 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.408897    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:26.408971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:26.408930    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:27.219136 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:27.219102    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" event={"ID":"9b889198-e09b-4030-8c01-7a4565bbb81d","Type":"ContainerStarted","Data":"ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684"}
Jul 13 21:48:27.219511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:27.219165    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" event={"ID":"9b889198-e09b-4030-8c01-7a4565bbb81d","Type":"ContainerStarted","Data":"fbebf4a98edaeace10d43ea0412103dcee765485a70ff69985406c7cd97af0f1"}
Jul 13 21:48:27.239195 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:27.239147    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" podStartSLOduration=2.239132321 podStartE2EDuration="2.239132321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:48:27.237396458 +0000 UTC m=+2353.895053465" watchObservedRunningTime="2026-07-13 21:48:27.239132321 +0000 UTC m=+2353.896789327"
Jul 13 21:48:27.271452 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:27.271422    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:27.276406 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:27.276383    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:28.223690 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:28.223660    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:28.224478 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:28.224459    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:29.092243 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.092206    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:29.161218 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.161187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:29.165423 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.165398    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"]
Jul 13 21:48:29.165873 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.165852    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c37cc970-b251-40ac-8b25-eb7f36a68b76" containerName="istio-proxy"
Jul 13 21:48:29.169000 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.168979    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.171577 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.171557    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-509096-serving-cert\""
Jul 13 21:48:29.178476 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.178444    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"]
Jul 13 21:48:29.231075 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.231044    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:29.365794 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.365706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkgm\" (UniqueName: \"kubernetes.io/projected/84a298ac-6323-4ed7-80d0-23a914e1efe3-kube-api-access-zxkgm\") pod \"maas-api-e2e-api-a-509096-847b477448-dp8s8\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.365794 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.365759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a298ac-6323-4ed7-80d0-23a914e1efe3-maas-api-tls\") pod \"maas-api-e2e-api-a-509096-847b477448-dp8s8\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.365794 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.365784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a298ac-6323-4ed7-80d0-23a914e1efe3-openshift-service-ca\") pod \"maas-api-e2e-api-a-509096-847b477448-dp8s8\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.442435 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.442399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:29.466542 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.466519    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkgm\" (UniqueName: \"kubernetes.io/projected/84a298ac-6323-4ed7-80d0-23a914e1efe3-kube-api-access-zxkgm\") pod \"maas-api-e2e-api-a-509096-847b477448-dp8s8\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.466675 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.466552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a298ac-6323-4ed7-80d0-23a914e1efe3-maas-api-tls\") pod \"maas-api-e2e-api-a-509096-847b477448-dp8s8\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.466675 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.466575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a298ac-6323-4ed7-80d0-23a914e1efe3-openshift-service-ca\") pod \"maas-api-e2e-api-a-509096-847b477448-dp8s8\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.467321 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.467298    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a298ac-6323-4ed7-80d0-23a914e1efe3-openshift-service-ca\") pod \"maas-api-e2e-api-a-509096-847b477448-dp8s8\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.469163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.469131    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a298ac-6323-4ed7-80d0-23a914e1efe3-maas-api-tls\") pod \"maas-api-e2e-api-a-509096-847b477448-dp8s8\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.477697 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.477670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkgm\" (UniqueName: \"kubernetes.io/projected/84a298ac-6323-4ed7-80d0-23a914e1efe3-kube-api-access-zxkgm\") pod \"maas-api-e2e-api-a-509096-847b477448-dp8s8\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.480503 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.480478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:29.614806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.614783    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"]
Jul 13 21:48:29.616664 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:48:29.616575    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a298ac_6323_4ed7_80d0_23a914e1efe3.slice/crio-bf4b8bbe8654c61951dcd5fdb9c1256391abfd852fe2dd9bcae966c0da22fb44 WatchSource:0}: Error finding container bf4b8bbe8654c61951dcd5fdb9c1256391abfd852fe2dd9bcae966c0da22fb44: Status 404 returned error can't find the container with id bf4b8bbe8654c61951dcd5fdb9c1256391abfd852fe2dd9bcae966c0da22fb44
Jul 13 21:48:29.618018 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.617995    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:48:29.781720 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.781677    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:29.882027 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:29.881948    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:30.234713 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.234682    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8" event={"ID":"84a298ac-6323-4ed7-80d0-23a914e1efe3","Type":"ContainerStarted","Data":"e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a"}
Jul 13 21:48:30.235034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.234871    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:30.235034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.234898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8" event={"ID":"84a298ac-6323-4ed7-80d0-23a914e1efe3","Type":"ContainerStarted","Data":"bf4b8bbe8654c61951dcd5fdb9c1256391abfd852fe2dd9bcae966c0da22fb44"}
Jul 13 21:48:30.252831 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.252782    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8" podStartSLOduration=0.836180866 podStartE2EDuration="1.252766256s" totalImagesPullingTime="416.58539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:48:30.251637119 +0000 UTC m=+2356.909294126" watchObservedRunningTime="2026-07-13 21:48:30.252766256 +0000 UTC m=+2356.910423260"
Jul 13 21:48:30.368460 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.368433    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:48:30.373844 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.373820    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-envoy\") pod \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") "
Jul 13 21:48:30.373956 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.373892    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-credential-socket\") pod \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") "
Jul 13 21:48:30.373956 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.373931    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v7ft\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-kube-api-access-9v7ft\") pod \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") "
Jul 13 21:48:30.374061 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.373959    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-socket\") pod \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") "
Jul 13 21:48:30.374061 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.373982    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-podinfo\") pod \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") "
Jul 13 21:48:30.374061 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.374051    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istiod-ca-cert\") pod \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") "
Jul 13 21:48:30.374220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.374113    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-certs\") pod \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") "
Jul 13 21:48:30.374220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.374147    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-token\") pod \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") "
Jul 13 21:48:30.374220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.374173    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-data\") pod \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\" (UID: \"cb80a2cb-0a28-4949-b448-2c9bf4df36d9\") "
Jul 13 21:48:30.374372 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.374352    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb80a2cb-0a28-4949-b448-2c9bf4df36d9" (UID: "cb80a2cb-0a28-4949-b448-2c9bf4df36d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:30.374475 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.374373    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb80a2cb-0a28-4949-b448-2c9bf4df36d9" (UID: "cb80a2cb-0a28-4949-b448-2c9bf4df36d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:30.374544 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.374437    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:30.374792 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.374752    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb80a2cb-0a28-4949-b448-2c9bf4df36d9" (UID: "cb80a2cb-0a28-4949-b448-2c9bf4df36d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:48:30.375042 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.375009    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb80a2cb-0a28-4949-b448-2c9bf4df36d9" (UID: "cb80a2cb-0a28-4949-b448-2c9bf4df36d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:30.375222 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.375185    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb80a2cb-0a28-4949-b448-2c9bf4df36d9" (UID: "cb80a2cb-0a28-4949-b448-2c9bf4df36d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:30.376324 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.376275    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-kube-api-access-9v7ft" (OuterVolumeSpecName: "kube-api-access-9v7ft") pod "cb80a2cb-0a28-4949-b448-2c9bf4df36d9" (UID: "cb80a2cb-0a28-4949-b448-2c9bf4df36d9"). InnerVolumeSpecName "kube-api-access-9v7ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:48:30.376577 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.376556    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb80a2cb-0a28-4949-b448-2c9bf4df36d9" (UID: "cb80a2cb-0a28-4949-b448-2c9bf4df36d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:48:30.376733 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.376718    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb80a2cb-0a28-4949-b448-2c9bf4df36d9" (UID: "cb80a2cb-0a28-4949-b448-2c9bf4df36d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:48:30.376999 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.376982    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb80a2cb-0a28-4949-b448-2c9bf4df36d9" (UID: "cb80a2cb-0a28-4949-b448-2c9bf4df36d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:48:30.475680 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.475647    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:30.475680 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.475678    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:30.475877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.475691    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:30.475877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.475698    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:30.475877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.475708    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:30.475877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.475720    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v7ft\" (UniqueName: \"kubernetes.io/projected/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-kube-api-access-9v7ft\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:30.475877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.475729    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:30.475877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:30.475738    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80a2cb-0a28-4949-b448-2c9bf4df36d9-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:48:31.243849 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.243818    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerID="459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec" exitCode=0
Jul 13 21:48:31.244287 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.243868    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"
Jul 13 21:48:31.244287 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.243903    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" event={"ID":"cb80a2cb-0a28-4949-b448-2c9bf4df36d9","Type":"ContainerDied","Data":"459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec"}
Jul 13 21:48:31.244287 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.243957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85" event={"ID":"cb80a2cb-0a28-4949-b448-2c9bf4df36d9","Type":"ContainerDied","Data":"4e7a31defcb145d19668df29c402d05d9362d0f74c6e362a83b3fb7153d68a3e"}
Jul 13 21:48:31.244287 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.243980    2564 scope.go:117] "RemoveContainer" containerID="459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec"
Jul 13 21:48:31.253225 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.253210    2564 scope.go:117] "RemoveContainer" containerID="ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f"
Jul 13 21:48:31.261060 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.261039    2564 scope.go:117] "RemoveContainer" containerID="459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec"
Jul 13 21:48:31.261290 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:48:31.261272    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec\": container with ID starting with 459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec not found: ID does not exist" containerID="459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec"
Jul 13 21:48:31.261339 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.261300    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec"} err="failed to get container status \"459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec\": rpc error: code = NotFound desc = could not find container \"459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec\": container with ID starting with 459eefbf27423043273039032ea1edfa18183a51c80fe0e8a17b64d1c0ed8cec not found: ID does not exist"
Jul 13 21:48:31.261339 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.261316    2564 scope.go:117] "RemoveContainer" containerID="ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f"
Jul 13 21:48:31.261563 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:48:31.261547    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f\": container with ID starting with ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f not found: ID does not exist" containerID="ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f"
Jul 13 21:48:31.261652 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.261569    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f"} err="failed to get container status \"ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f\": rpc error: code = NotFound desc = could not find container \"ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f\": container with ID starting with ad4789740e35d8df5bcef06b233084ecf00b86ff43a3c9437e09426f9162c12f not found: ID does not exist"
Jul 13 21:48:31.269804 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.269783    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"]
Jul 13 21:48:31.273074 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.273053    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2522e6-openshift-default-5fd8b5d655-7fd85"]
Jul 13 21:48:31.821175 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:31.821140    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" path="/var/lib/kubelet/pods/cb80a2cb-0a28-4949-b448-2c9bf4df36d9/volumes"
Jul 13 21:48:33.598957 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:33.598923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:34.011833 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.011781    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:34.272702 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.272627    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"]
Jul 13 21:48:34.272934 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.272922    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy"
Jul 13 21:48:34.272999 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.272935    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy"
Jul 13 21:48:34.272999 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.272994    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy"
Jul 13 21:48:34.273107 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.273003    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy"
Jul 13 21:48:34.277423 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.277397    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.280715 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.280694    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-006a57-openshift-default-dockercfg-htx7h\""
Jul 13 21:48:34.292729 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.292691    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"]
Jul 13 21:48:34.301468 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.301438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5bfd\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-kube-api-access-g5bfd\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.301618 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.301509    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-socket\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.301618 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.301540    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-istio-token\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.301618 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.301573    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32add035-3fdd-4f9c-a800-40802ac6839d-istio-podinfo\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.301759 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.301672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-envoy\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.301759 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.301706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-credential-socket\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.301759 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.301740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-data\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.301910 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.301774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-certs\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.301910 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.301827    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32add035-3fdd-4f9c-a800-40802ac6839d-istiod-ca-cert\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.305333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.305303    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:34.403088 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-envoy\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403088 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-credential-socket\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403289 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-data\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403289 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-certs\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403289 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32add035-3fdd-4f9c-a800-40802ac6839d-istiod-ca-cert\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403451 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5bfd\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-kube-api-access-g5bfd\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403451 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-socket\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403451 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-istio-token\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403451 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-credential-socket\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403451 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32add035-3fdd-4f9c-a800-40802ac6839d-istio-podinfo\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403741 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403545    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-data\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403741 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403692    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-certs\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.403966 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.403945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-socket\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.404251 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.404227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32add035-3fdd-4f9c-a800-40802ac6839d-istiod-ca-cert\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.405508 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.405483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-envoy\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.405969 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.405954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32add035-3fdd-4f9c-a800-40802ac6839d-istio-podinfo\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.411101 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.411073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-istio-token\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.411332 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.411304    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5bfd\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-kube-api-access-g5bfd\") pod \"e2e-api-b-006a57-openshift-default-66cf9b4575-772vt\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") " pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.589615 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.589537    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:34.723639 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.723603    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"]
Jul 13 21:48:34.724876 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.724851    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:34.726878 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:48:34.726853    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32add035_3fdd_4f9c_a800_40802ac6839d.slice/crio-bf8f0a8fdd110a7155ab4fd837bc4732b3ad36c391eea78d52753163960119c2 WatchSource:0}: Error finding container bf8f0a8fdd110a7155ab4fd837bc4732b3ad36c391eea78d52753163960119c2: Status 404 returned error can't find the container with id bf8f0a8fdd110a7155ab4fd837bc4732b3ad36c391eea78d52753163960119c2
Jul 13 21:48:34.729424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.729393    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:34.729491 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.729461    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:34.729547 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:34.729496    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:35.264877 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:35.264837    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" event={"ID":"32add035-3fdd-4f9c-a800-40802ac6839d","Type":"ContainerStarted","Data":"713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c"}
Jul 13 21:48:35.265036 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:35.264900    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" event={"ID":"32add035-3fdd-4f9c-a800-40802ac6839d","Type":"ContainerStarted","Data":"bf8f0a8fdd110a7155ab4fd837bc4732b3ad36c391eea78d52753163960119c2"}
Jul 13 21:48:35.289076 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:35.289034    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" podStartSLOduration=1.289018625 podStartE2EDuration="1.289018625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:48:35.287278688 +0000 UTC m=+2361.944935694" watchObservedRunningTime="2026-07-13 21:48:35.289018625 +0000 UTC m=+2361.946675630"
Jul 13 21:48:35.590451 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:35.590363    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:35.595037 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:35.595012    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:36.257924 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:36.257883    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:48:36.271552 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:36.271527    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:36.272428 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:36.272405    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:37.191371 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.191326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:37.273511 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.273477    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"]
Jul 13 21:48:37.274011 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.273977    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy"
Jul 13 21:48:37.274011 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.273997    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb80a2cb-0a28-4949-b448-2c9bf4df36d9" containerName="istio-proxy"
Jul 13 21:48:37.274976 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.274948    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:37.277745 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.277726    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.280496 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.280475    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-006a57-serving-cert\""
Jul 13 21:48:37.289226 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.289204    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"]
Jul 13 21:48:37.328459 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.328432    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d98022-458d-4744-9139-a471fd5dee14-maas-api-tls\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.328564 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.328467    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d98022-458d-4744-9139-a471fd5dee14-openshift-service-ca\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.328564 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.328511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csw8r\" (UniqueName: \"kubernetes.io/projected/e4d98022-458d-4744-9139-a471fd5dee14-kube-api-access-csw8r\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.340938 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.340911    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:37.430208 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.430176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d98022-458d-4744-9139-a471fd5dee14-maas-api-tls\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.430392 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.430224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d98022-458d-4744-9139-a471fd5dee14-openshift-service-ca\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.430392 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:48:37.430321    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-serving-cert: secret "maas-api-e2e-api-b-006a57-serving-cert" not found
Jul 13 21:48:37.430392 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.430347    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csw8r\" (UniqueName: \"kubernetes.io/projected/e4d98022-458d-4744-9139-a471fd5dee14-kube-api-access-csw8r\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.430392 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:48:37.430379    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d98022-458d-4744-9139-a471fd5dee14-maas-api-tls podName:e4d98022-458d-4744-9139-a471fd5dee14 nodeName:}" failed. No retries permitted until 2026-07-13 21:48:37.930363291 +0000 UTC m=+2364.588020278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4d98022-458d-4744-9139-a471fd5dee14-maas-api-tls") pod "maas-api-e2e-api-b-006a57-64cf4b5689-gd79m" (UID: "e4d98022-458d-4744-9139-a471fd5dee14") : secret "maas-api-e2e-api-b-006a57-serving-cert" not found
Jul 13 21:48:37.430851 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.430835    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d98022-458d-4744-9139-a471fd5dee14-openshift-service-ca\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.441531 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.441466    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csw8r\" (UniqueName: \"kubernetes.io/projected/e4d98022-458d-4744-9139-a471fd5dee14-kube-api-access-csw8r\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.613726 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.613688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:37.934761 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.934728    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d98022-458d-4744-9139-a471fd5dee14-maas-api-tls\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.937420 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.937364    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d98022-458d-4744-9139-a471fd5dee14-maas-api-tls\") pod \"maas-api-e2e-api-b-006a57-64cf4b5689-gd79m\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:37.937614 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:37.937544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:38.038626 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:38.038578    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:38.188980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:38.188894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:38.371611 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:38.370934    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"]
Jul 13 21:48:38.377036 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:48:38.377007    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d98022_458d_4744_9139_a471fd5dee14.slice/crio-02f9a124b9bc3a3444d864c117e728e7518c3e6f16ef023a74716d9bc5bd05a1 WatchSource:0}: Error finding container 02f9a124b9bc3a3444d864c117e728e7518c3e6f16ef023a74716d9bc5bd05a1: Status 404 returned error can't find the container with id 02f9a124b9bc3a3444d864c117e728e7518c3e6f16ef023a74716d9bc5bd05a1
Jul 13 21:48:38.799689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:38.799643    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:39.289958 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:39.289925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m" event={"ID":"e4d98022-458d-4744-9139-a471fd5dee14","Type":"ContainerStarted","Data":"4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3"}
Jul 13 21:48:39.290127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:39.289972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m" event={"ID":"e4d98022-458d-4744-9139-a471fd5dee14","Type":"ContainerStarted","Data":"02f9a124b9bc3a3444d864c117e728e7518c3e6f16ef023a74716d9bc5bd05a1"}
Jul 13 21:48:39.290127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:39.290114    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:39.305860 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:39.305817    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m" podStartSLOduration=1.824907362 podStartE2EDuration="2.305803441s" totalImagesPullingTime="480.896079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:48:39.304448371 +0000 UTC m=+2365.962105377" watchObservedRunningTime="2026-07-13 21:48:39.305803441 +0000 UTC m=+2365.963460453"
Jul 13 21:48:41.657202 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:41.657160    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:43.276685 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:43.276625    2564 patch_prober.go:28] interesting pod/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:42474->10.132.0.55:15021: read: connection reset by peer" start-of-body=
Jul 13 21:48:43.277019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:43.276684    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:42474->10.132.0.55:15021: read: connection reset by peer"
Jul 13 21:48:43.313575 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:43.313547    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerID="ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684" exitCode=0
Jul 13 21:48:43.313704 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:43.313627    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" event={"ID":"9b889198-e09b-4030-8c01-7a4565bbb81d","Type":"ContainerDied","Data":"ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684"}
Jul 13 21:48:43.314116 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:43.314098    2564 scope.go:117] "RemoveContainer" containerID="ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684"
Jul 13 21:48:43.315349 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:43.315317    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:43.315425 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:43.315404    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:43.315463 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:43.315445    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:44.320554 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:44.320520    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" event={"ID":"9b889198-e09b-4030-8c01-7a4565bbb81d","Type":"ContainerStarted","Data":"7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab"}
Jul 13 21:48:45.271163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:45.271121    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:45.271349 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:45.271247    2564 patch_prober.go:28] interesting pod/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 13 21:48:45.271349 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:45.271293    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 13 21:48:45.301452 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:45.301426    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:48:46.271706 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:46.271666    2564 patch_prober.go:28] interesting pod/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 13 21:48:46.272092 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:46.271734    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 13 21:48:47.271420 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:47.271381    2564 patch_prober.go:28] interesting pod/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 13 21:48:47.271603 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:47.271452    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 13 21:48:48.275558 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:48.275525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:48.276104 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:48.276086    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:48.276808 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:48.276790    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:48:49.348250 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:49.348218    2564 generic.go:356] "Generic (PLEG): container finished" podID="32add035-3fdd-4f9c-a800-40802ac6839d" containerID="713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c" exitCode=0
Jul 13 21:48:49.348785 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:49.348296    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" event={"ID":"32add035-3fdd-4f9c-a800-40802ac6839d","Type":"ContainerDied","Data":"713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c"}
Jul 13 21:48:49.348785 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:49.348743    2564 scope.go:117] "RemoveContainer" containerID="713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c"
Jul 13 21:48:49.349701 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:49.349674    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:49.349795 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:49.349737    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:49.349795 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:49.349767    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:48:50.112542 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:50.112514    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:50.212971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:50.212940    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:50.356345 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:50.356312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" event={"ID":"32add035-3fdd-4f9c-a800-40802ac6839d","Type":"ContainerStarted","Data":"c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92"}
Jul 13 21:48:50.590363 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:50.590329    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:50.591386 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:50.591357    2564 patch_prober.go:28] interesting pod/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 13 21:48:50.591516 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:50.591416    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 13 21:48:51.272471 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:51.272430    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:51.590820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:51.590738    2564 patch_prober.go:28] interesting pod/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 13 21:48:51.590820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:51.590791    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 13 21:48:52.590109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:52.590070    2564 patch_prober.go:28] interesting pod/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 13 21:48:52.590293 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:52.590129    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 13 21:48:53.589949 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:53.589915    2564 patch_prober.go:28] interesting pod/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 13 21:48:53.590317 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:53.589972    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 13 21:48:54.594452 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:54.594412    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:54.595265 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:54.595245    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:48:59.870079 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:59.870048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:48:59.971951 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:48:59.971923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:00.098408 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.098375    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"]
Jul 13 21:49:00.098639 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.098577    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8" podUID="84a298ac-6323-4ed7-80d0-23a914e1efe3" containerName="maas-api" containerID="cri-o://e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a" gracePeriod=30
Jul 13 21:49:00.117508 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.117472    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:00.353758 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.353734    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:49:00.410846 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.410811    2564 generic.go:356] "Generic (PLEG): container finished" podID="84a298ac-6323-4ed7-80d0-23a914e1efe3" containerID="e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a" exitCode=0
Jul 13 21:49:00.410975 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.410864    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"
Jul 13 21:49:00.410975 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.410928    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8" event={"ID":"84a298ac-6323-4ed7-80d0-23a914e1efe3","Type":"ContainerDied","Data":"e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a"}
Jul 13 21:49:00.411129 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.411025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8" event={"ID":"84a298ac-6323-4ed7-80d0-23a914e1efe3","Type":"ContainerDied","Data":"bf4b8bbe8654c61951dcd5fdb9c1256391abfd852fe2dd9bcae966c0da22fb44"}
Jul 13 21:49:00.411129 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.411053    2564 scope.go:117] "RemoveContainer" containerID="e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a"
Jul 13 21:49:00.419855 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.419839    2564 scope.go:117] "RemoveContainer" containerID="e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a"
Jul 13 21:49:00.420073 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:49:00.420049    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a\": container with ID starting with e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a not found: ID does not exist" containerID="e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a"
Jul 13 21:49:00.420167 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.420078    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a"} err="failed to get container status \"e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a\": rpc error: code = NotFound desc = could not find container \"e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a\": container with ID starting with e2b17b8bed1e702a1f6f0f16d375f0187afb1df0d10df7f855c1bc03bda7366a not found: ID does not exist"
Jul 13 21:49:00.440370 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.440350    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a298ac-6323-4ed7-80d0-23a914e1efe3-maas-api-tls\") pod \"84a298ac-6323-4ed7-80d0-23a914e1efe3\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") "
Jul 13 21:49:00.440469 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.440390    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxkgm\" (UniqueName: \"kubernetes.io/projected/84a298ac-6323-4ed7-80d0-23a914e1efe3-kube-api-access-zxkgm\") pod \"84a298ac-6323-4ed7-80d0-23a914e1efe3\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") "
Jul 13 21:49:00.440469 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.440409    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a298ac-6323-4ed7-80d0-23a914e1efe3-openshift-service-ca\") pod \"84a298ac-6323-4ed7-80d0-23a914e1efe3\" (UID: \"84a298ac-6323-4ed7-80d0-23a914e1efe3\") "
Jul 13 21:49:00.440806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.440780    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a298ac-6323-4ed7-80d0-23a914e1efe3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84a298ac-6323-4ed7-80d0-23a914e1efe3" (UID: "84a298ac-6323-4ed7-80d0-23a914e1efe3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:49:00.442622 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.442597    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a298ac-6323-4ed7-80d0-23a914e1efe3-kube-api-access-zxkgm" (OuterVolumeSpecName: "kube-api-access-zxkgm") pod "84a298ac-6323-4ed7-80d0-23a914e1efe3" (UID: "84a298ac-6323-4ed7-80d0-23a914e1efe3"). InnerVolumeSpecName "kube-api-access-zxkgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:49:00.442712 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.442620    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a298ac-6323-4ed7-80d0-23a914e1efe3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84a298ac-6323-4ed7-80d0-23a914e1efe3" (UID: "84a298ac-6323-4ed7-80d0-23a914e1efe3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:49:00.541032 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.541005    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a298ac-6323-4ed7-80d0-23a914e1efe3-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:00.541032 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.541030    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a298ac-6323-4ed7-80d0-23a914e1efe3-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:00.541196 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.541039    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxkgm\" (UniqueName: \"kubernetes.io/projected/84a298ac-6323-4ed7-80d0-23a914e1efe3-kube-api-access-zxkgm\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:00.731810 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.731777    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"]
Jul 13 21:49:00.737264 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:00.737243    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-509096-847b477448-dp8s8"]
Jul 13 21:49:01.822442 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:01.822397    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84a298ac-6323-4ed7-80d0-23a914e1efe3" path="/var/lib/kubelet/pods/84a298ac-6323-4ed7-80d0-23a914e1efe3/volumes"
Jul 13 21:49:04.589468 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:04.589439    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:10.298256 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:10.298225    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:10.399772 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:10.399710    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:13.949398 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:13.949372    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:49:13.955696 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:13.955675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:49:15.025614 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:15.025555    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:15.225394 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:15.225363    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"]
Jul 13 21:49:15.225627 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:15.225569    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy" containerID="cri-o://7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab" gracePeriod=30
Jul 13 21:49:15.254549 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:15.254522    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:15.495650 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:15.495614    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:18.278653 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:18.278617    2564 patch_prober.go:28] interesting pod/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 21:49:18.279087 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:18.278682    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:49:20.486078 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.486021    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:49:20.507257 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.507077    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerID="7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab" exitCode=0
Jul 13 21:49:20.507257 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.507139    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"
Jul 13 21:49:20.507257 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.507146    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" event={"ID":"9b889198-e09b-4030-8c01-7a4565bbb81d","Type":"ContainerDied","Data":"7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab"}
Jul 13 21:49:20.507257 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.507252    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg" event={"ID":"9b889198-e09b-4030-8c01-7a4565bbb81d","Type":"ContainerDied","Data":"fbebf4a98edaeace10d43ea0412103dcee765485a70ff69985406c7cd97af0f1"}
Jul 13 21:49:20.507637 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.507290    2564 scope.go:117] "RemoveContainer" containerID="7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab"
Jul 13 21:49:20.515790 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.515770    2564 scope.go:117] "RemoveContainer" containerID="ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684"
Jul 13 21:49:20.523421 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.523406    2564 scope.go:117] "RemoveContainer" containerID="7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab"
Jul 13 21:49:20.523702 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:49:20.523682    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab\": container with ID starting with 7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab not found: ID does not exist" containerID="7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab"
Jul 13 21:49:20.523796 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.523710    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab"} err="failed to get container status \"7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab\": rpc error: code = NotFound desc = could not find container \"7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab\": container with ID starting with 7fbda649c6d2cab6718413840c7b1e94a76b1bc40d0c3aaec59ce3a5797cbfab not found: ID does not exist"
Jul 13 21:49:20.523796 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.523732    2564 scope.go:117] "RemoveContainer" containerID="ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684"
Jul 13 21:49:20.524017 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:49:20.523997    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684\": container with ID starting with ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684 not found: ID does not exist" containerID="ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684"
Jul 13 21:49:20.524073 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.524030    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684"} err="failed to get container status \"ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684\": rpc error: code = NotFound desc = could not find container \"ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684\": container with ID starting with ca9e28b10e0fe47347f96b50763aee105f759d54fb673ed8d690096307acc684 not found: ID does not exist"
Jul 13 21:49:20.590721 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.590700    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-certs\") pod \"9b889198-e09b-4030-8c01-7a4565bbb81d\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") "
Jul 13 21:49:20.590826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.590733    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-token\") pod \"9b889198-e09b-4030-8c01-7a4565bbb81d\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") "
Jul 13 21:49:20.590826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.590752    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b889198-e09b-4030-8c01-7a4565bbb81d-istiod-ca-cert\") pod \"9b889198-e09b-4030-8c01-7a4565bbb81d\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") "
Jul 13 21:49:20.590826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.590768    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-socket\") pod \"9b889198-e09b-4030-8c01-7a4565bbb81d\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") "
Jul 13 21:49:20.590826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.590791    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45pc5\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-kube-api-access-45pc5\") pod \"9b889198-e09b-4030-8c01-7a4565bbb81d\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") "
Jul 13 21:49:20.591034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.590831    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-credential-socket\") pod \"9b889198-e09b-4030-8c01-7a4565bbb81d\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") "
Jul 13 21:49:20.591034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.590855    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-data\") pod \"9b889198-e09b-4030-8c01-7a4565bbb81d\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") "
Jul 13 21:49:20.591034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.590890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-podinfo\") pod \"9b889198-e09b-4030-8c01-7a4565bbb81d\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") "
Jul 13 21:49:20.591034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.590958    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-envoy\") pod \"9b889198-e09b-4030-8c01-7a4565bbb81d\" (UID: \"9b889198-e09b-4030-8c01-7a4565bbb81d\") "
Jul 13 21:49:20.591034 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.591011    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b889198-e09b-4030-8c01-7a4565bbb81d" (UID: "9b889198-e09b-4030-8c01-7a4565bbb81d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:20.591295 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.591208    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b889198-e09b-4030-8c01-7a4565bbb81d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b889198-e09b-4030-8c01-7a4565bbb81d" (UID: "9b889198-e09b-4030-8c01-7a4565bbb81d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:49:20.591295 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.591247    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:20.591405 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.591303    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b889198-e09b-4030-8c01-7a4565bbb81d" (UID: "9b889198-e09b-4030-8c01-7a4565bbb81d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:20.591697 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.591430    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b889198-e09b-4030-8c01-7a4565bbb81d" (UID: "9b889198-e09b-4030-8c01-7a4565bbb81d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:20.592032 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.592002    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b889198-e09b-4030-8c01-7a4565bbb81d" (UID: "9b889198-e09b-4030-8c01-7a4565bbb81d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:20.593126 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.593102    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b889198-e09b-4030-8c01-7a4565bbb81d" (UID: "9b889198-e09b-4030-8c01-7a4565bbb81d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:49:20.593229 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.593149    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b889198-e09b-4030-8c01-7a4565bbb81d" (UID: "9b889198-e09b-4030-8c01-7a4565bbb81d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:49:20.593275 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.593239    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b889198-e09b-4030-8c01-7a4565bbb81d" (UID: "9b889198-e09b-4030-8c01-7a4565bbb81d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:20.593275 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.593268    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-kube-api-access-45pc5" (OuterVolumeSpecName: "kube-api-access-45pc5") pod "9b889198-e09b-4030-8c01-7a4565bbb81d" (UID: "9b889198-e09b-4030-8c01-7a4565bbb81d"). InnerVolumeSpecName "kube-api-access-45pc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:49:20.692500 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.692474    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:20.692500 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.692495    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:20.692500 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.692504    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:20.692724 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.692513    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b889198-e09b-4030-8c01-7a4565bbb81d-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:20.692724 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.692520    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:20.692724 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.692529    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45pc5\" (UniqueName: \"kubernetes.io/projected/9b889198-e09b-4030-8c01-7a4565bbb81d-kube-api-access-45pc5\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:20.692724 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.692539    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:20.692724 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.692547    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b889198-e09b-4030-8c01-7a4565bbb81d-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:20.828348 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.828317    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"]
Jul 13 21:49:20.831884 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:20.831862    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-509096-openshift-default-59d6fff597-sp5xg"]
Jul 13 21:49:21.822366 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:21.822327    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" path="/var/lib/kubelet/pods/9b889198-e09b-4030-8c01-7a4565bbb81d/volumes"
Jul 13 21:49:26.086901 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.083967    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"]
Jul 13 21:49:26.086901 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.084426    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m" podUID="e4d98022-458d-4744-9139-a471fd5dee14" containerName="maas-api" containerID="cri-o://4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3" gracePeriod=30
Jul 13 21:49:26.112858 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.112823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:26.335542 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.335520    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:49:26.435869 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.435800    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d98022-458d-4744-9139-a471fd5dee14-maas-api-tls\") pod \"e4d98022-458d-4744-9139-a471fd5dee14\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") "
Jul 13 21:49:26.435869 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.435843    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d98022-458d-4744-9139-a471fd5dee14-openshift-service-ca\") pod \"e4d98022-458d-4744-9139-a471fd5dee14\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") "
Jul 13 21:49:26.436045 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.435894    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csw8r\" (UniqueName: \"kubernetes.io/projected/e4d98022-458d-4744-9139-a471fd5dee14-kube-api-access-csw8r\") pod \"e4d98022-458d-4744-9139-a471fd5dee14\" (UID: \"e4d98022-458d-4744-9139-a471fd5dee14\") "
Jul 13 21:49:26.436228 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.436186    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d98022-458d-4744-9139-a471fd5dee14-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4d98022-458d-4744-9139-a471fd5dee14" (UID: "e4d98022-458d-4744-9139-a471fd5dee14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:49:26.438046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.438021    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d98022-458d-4744-9139-a471fd5dee14-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4d98022-458d-4744-9139-a471fd5dee14" (UID: "e4d98022-458d-4744-9139-a471fd5dee14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:49:26.438128 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.438061    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d98022-458d-4744-9139-a471fd5dee14-kube-api-access-csw8r" (OuterVolumeSpecName: "kube-api-access-csw8r") pod "e4d98022-458d-4744-9139-a471fd5dee14" (UID: "e4d98022-458d-4744-9139-a471fd5dee14"). InnerVolumeSpecName "kube-api-access-csw8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:49:26.536422 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.536399    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csw8r\" (UniqueName: \"kubernetes.io/projected/e4d98022-458d-4744-9139-a471fd5dee14-kube-api-access-csw8r\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:26.536422 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.536422    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d98022-458d-4744-9139-a471fd5dee14-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:26.536570 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.536431    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d98022-458d-4744-9139-a471fd5dee14-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:26.540514 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.540491    2564 generic.go:356] "Generic (PLEG): container finished" podID="e4d98022-458d-4744-9139-a471fd5dee14" containerID="4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3" exitCode=0
Jul 13 21:49:26.540627 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.540525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m" event={"ID":"e4d98022-458d-4744-9139-a471fd5dee14","Type":"ContainerDied","Data":"4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3"}
Jul 13 21:49:26.540627 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.540533    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"
Jul 13 21:49:26.540627 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.540562    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m" event={"ID":"e4d98022-458d-4744-9139-a471fd5dee14","Type":"ContainerDied","Data":"02f9a124b9bc3a3444d864c117e728e7518c3e6f16ef023a74716d9bc5bd05a1"}
Jul 13 21:49:26.540627 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.540608    2564 scope.go:117] "RemoveContainer" containerID="4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3"
Jul 13 21:49:26.549687 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.549671    2564 scope.go:117] "RemoveContainer" containerID="4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3"
Jul 13 21:49:26.549934 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:49:26.549917    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3\": container with ID starting with 4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3 not found: ID does not exist" containerID="4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3"
Jul 13 21:49:26.549991 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.549942    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3"} err="failed to get container status \"4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3\": rpc error: code = NotFound desc = could not find container \"4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3\": container with ID starting with 4393c92c33ac508d1da01558f40d6b943991618ec7bad76bdbfd5042d82c2ea3 not found: ID does not exist"
Jul 13 21:49:26.560749 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.560728    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"]
Jul 13 21:49:26.565271 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.565251    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-006a57-64cf4b5689-gd79m"]
Jul 13 21:49:26.589205 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:26.589177    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:27.821535 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:27.821500    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d98022-458d-4744-9139-a471fd5dee14" path="/var/lib/kubelet/pods/e4d98022-458d-4744-9139-a471fd5dee14/volumes"
Jul 13 21:49:30.687418 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:30.687380    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:36.253441 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:36.253354    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:36.316811 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:36.316768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:41.310430 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:41.310389    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"]
Jul 13 21:49:41.310948 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:41.310670    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy" containerID="cri-o://c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92" gracePeriod=30
Jul 13 21:49:41.330854 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:41.330804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:41.546928 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:41.546895    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:41.864120 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:41.864087    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:42.134362 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134291    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"]
Jul 13 21:49:42.134710 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134692    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d98022-458d-4744-9139-a471fd5dee14" containerName="maas-api"
Jul 13 21:49:42.134710 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134714    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d98022-458d-4744-9139-a471fd5dee14" containerName="maas-api"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134725    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134731    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134746    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134752    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134766    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a298ac-6323-4ed7-80d0-23a914e1efe3" containerName="maas-api"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134771    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a298ac-6323-4ed7-80d0-23a914e1efe3" containerName="maas-api"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134843    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134852    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b889198-e09b-4030-8c01-7a4565bbb81d" containerName="istio-proxy"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134859    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a298ac-6323-4ed7-80d0-23a914e1efe3" containerName="maas-api"
Jul 13 21:49:42.134880 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.134868    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d98022-458d-4744-9139-a471fd5dee14" containerName="maas-api"
Jul 13 21:49:42.138333 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.138312    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.140560 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.140542    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-8e2974-openshift-default-dockercfg-zw5tq\""
Jul 13 21:49:42.148771 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.148751    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"]
Jul 13 21:49:42.157354 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.157332    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:42.238474 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.238447    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-socket\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.238656 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.238483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-credential-socket\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.238656 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.238520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jnm7\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-kube-api-access-6jnm7\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.238656 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.238620    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-data\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.238656 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.238650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-istio-token\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.238834 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.238674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-certs\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.238834 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.238714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53133c74-3849-4ccb-8119-b2af7357b402-istiod-ca-cert\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.238834 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.238744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53133c74-3849-4ccb-8119-b2af7357b402-istio-podinfo\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.238834 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.238796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-envoy\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.339954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.339922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-envoy\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.339970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-socket\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340092    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-credential-socket\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jnm7\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-kube-api-access-6jnm7\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-data\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-istio-token\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-certs\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53133c74-3849-4ccb-8119-b2af7357b402-istiod-ca-cert\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53133c74-3849-4ccb-8119-b2af7357b402-istio-podinfo\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340775 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340345    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-socket\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340775 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-credential-socket\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340775 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340611    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-data\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.340890 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.340720    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-certs\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.341223 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.341202    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53133c74-3849-4ccb-8119-b2af7357b402-istiod-ca-cert\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.342469 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.342450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-envoy\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.342935 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.342917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53133c74-3849-4ccb-8119-b2af7357b402-istio-podinfo\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.352229 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.352203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-istio-token\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.352352 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.352242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jnm7\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-kube-api-access-6jnm7\") pod \"e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") " pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.414919 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.414851    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:42.448978 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.448953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:42.586795 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.586770    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"]
Jul 13 21:49:42.588204 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:49:42.588178    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53133c74_3849_4ccb_8119_b2af7357b402.slice/crio-e608df0ed1a32d984795661180a5beea9620343ff4136ff78be178ec3b12c24e WatchSource:0}: Error finding container e608df0ed1a32d984795661180a5beea9620343ff4136ff78be178ec3b12c24e: Status 404 returned error can't find the container with id e608df0ed1a32d984795661180a5beea9620343ff4136ff78be178ec3b12c24e
Jul 13 21:49:42.590557 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.590521    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:49:42.590674 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.590622    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:49:42.590674 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.590666    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:49:42.626047 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:42.626020    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" event={"ID":"53133c74-3849-4ccb-8119-b2af7357b402","Type":"ContainerStarted","Data":"e608df0ed1a32d984795661180a5beea9620343ff4136ff78be178ec3b12c24e"}
Jul 13 21:49:43.632344 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:43.632304    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" event={"ID":"53133c74-3849-4ccb-8119-b2af7357b402","Type":"ContainerStarted","Data":"a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b"}
Jul 13 21:49:43.651380 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:43.651303    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" podStartSLOduration=1.651284978 podStartE2EDuration="1.651284978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:49:43.650663756 +0000 UTC m=+2430.308320765" watchObservedRunningTime="2026-07-13 21:49:43.651284978 +0000 UTC m=+2430.308941986"
Jul 13 21:49:44.449925 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:44.449898    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:44.454722 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:44.454698    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:44.637315 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:44.637288    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:44.638237 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:44.638215    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:45.520899 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.520859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:45.587974 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.587943    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:45.589540 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.589511    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"]
Jul 13 21:49:45.593183 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.593165    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.595506 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.595488    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-8e2974-serving-cert\""
Jul 13 21:49:45.601766 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.601744    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"]
Jul 13 21:49:45.658535 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.658504    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:45.661840 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.661819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aae361-16ab-4b3d-a1d9-6902e56a4570-maas-api-tls\") pod \"maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.661949 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.661897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc7lz\" (UniqueName: \"kubernetes.io/projected/c2aae361-16ab-4b3d-a1d9-6902e56a4570-kube-api-access-dc7lz\") pod \"maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.661949 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.661931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aae361-16ab-4b3d-a1d9-6902e56a4570-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.759048 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.759020    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:45.762287 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.762265    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc7lz\" (UniqueName: \"kubernetes.io/projected/c2aae361-16ab-4b3d-a1d9-6902e56a4570-kube-api-access-dc7lz\") pod \"maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.762375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.762301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aae361-16ab-4b3d-a1d9-6902e56a4570-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.762486 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.762468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aae361-16ab-4b3d-a1d9-6902e56a4570-maas-api-tls\") pod \"maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.762948 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.762925    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aae361-16ab-4b3d-a1d9-6902e56a4570-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.764943 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.764917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aae361-16ab-4b3d-a1d9-6902e56a4570-maas-api-tls\") pod \"maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.770084 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.770056    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc7lz\" (UniqueName: \"kubernetes.io/projected/c2aae361-16ab-4b3d-a1d9-6902e56a4570-kube-api-access-dc7lz\") pod \"maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:45.906400 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:45.906307    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:46.057335 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.057308    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"]
Jul 13 21:49:46.059155 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:49:46.059125    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2aae361_16ab_4b3d_a1d9_6902e56a4570.slice/crio-84b95c18bc1e83516ffdde0ade0fa7d91d91e603e0e98ca3717912a8a940281a WatchSource:0}: Error finding container 84b95c18bc1e83516ffdde0ade0fa7d91d91e603e0e98ca3717912a8a940281a: Status 404 returned error can't find the container with id 84b95c18bc1e83516ffdde0ade0fa7d91d91e603e0e98ca3717912a8a940281a
Jul 13 21:49:46.161356 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.161276    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:46.264176 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.264123    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:46.555290 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.555268    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:49:46.568477 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568455    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-envoy\") pod \"32add035-3fdd-4f9c-a800-40802ac6839d\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") "
Jul 13 21:49:46.568601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568490    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32add035-3fdd-4f9c-a800-40802ac6839d-istio-podinfo\") pod \"32add035-3fdd-4f9c-a800-40802ac6839d\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") "
Jul 13 21:49:46.568601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568510    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-credential-socket\") pod \"32add035-3fdd-4f9c-a800-40802ac6839d\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") "
Jul 13 21:49:46.568601 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568537    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-data\") pod \"32add035-3fdd-4f9c-a800-40802ac6839d\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") "
Jul 13 21:49:46.568751 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568576    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5bfd\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-kube-api-access-g5bfd\") pod \"32add035-3fdd-4f9c-a800-40802ac6839d\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") "
Jul 13 21:49:46.568751 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568632    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-socket\") pod \"32add035-3fdd-4f9c-a800-40802ac6839d\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") "
Jul 13 21:49:46.568751 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568675    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32add035-3fdd-4f9c-a800-40802ac6839d-istiod-ca-cert\") pod \"32add035-3fdd-4f9c-a800-40802ac6839d\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") "
Jul 13 21:49:46.568751 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568713    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-istio-token\") pod \"32add035-3fdd-4f9c-a800-40802ac6839d\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") "
Jul 13 21:49:46.568751 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568749    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-certs\") pod \"32add035-3fdd-4f9c-a800-40802ac6839d\" (UID: \"32add035-3fdd-4f9c-a800-40802ac6839d\") "
Jul 13 21:49:46.569030 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568795    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32add035-3fdd-4f9c-a800-40802ac6839d" (UID: "32add035-3fdd-4f9c-a800-40802ac6839d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:46.569030 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.568996    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32add035-3fdd-4f9c-a800-40802ac6839d" (UID: "32add035-3fdd-4f9c-a800-40802ac6839d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:46.569030 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.569015    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:46.569195 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.569152    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32add035-3fdd-4f9c-a800-40802ac6839d" (UID: "32add035-3fdd-4f9c-a800-40802ac6839d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:46.569375 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.569113    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32add035-3fdd-4f9c-a800-40802ac6839d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32add035-3fdd-4f9c-a800-40802ac6839d" (UID: "32add035-3fdd-4f9c-a800-40802ac6839d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:49:46.569556 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.569531    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-data" (OuterVolumeSpecName: "istio-data") pod "32add035-3fdd-4f9c-a800-40802ac6839d" (UID: "32add035-3fdd-4f9c-a800-40802ac6839d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:46.570838 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.570814    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32add035-3fdd-4f9c-a800-40802ac6839d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32add035-3fdd-4f9c-a800-40802ac6839d" (UID: "32add035-3fdd-4f9c-a800-40802ac6839d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:49:46.570926 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.570861    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32add035-3fdd-4f9c-a800-40802ac6839d" (UID: "32add035-3fdd-4f9c-a800-40802ac6839d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:49:46.570926 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.570876    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-kube-api-access-g5bfd" (OuterVolumeSpecName: "kube-api-access-g5bfd") pod "32add035-3fdd-4f9c-a800-40802ac6839d" (UID: "32add035-3fdd-4f9c-a800-40802ac6839d"). InnerVolumeSpecName "kube-api-access-g5bfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:49:46.571112 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.571095    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-istio-token" (OuterVolumeSpecName: "istio-token") pod "32add035-3fdd-4f9c-a800-40802ac6839d" (UID: "32add035-3fdd-4f9c-a800-40802ac6839d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:49:46.648200 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.648171    2564 generic.go:356] "Generic (PLEG): container finished" podID="32add035-3fdd-4f9c-a800-40802ac6839d" containerID="c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92" exitCode=0
Jul 13 21:49:46.648330 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.648232    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"
Jul 13 21:49:46.648330 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.648255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" event={"ID":"32add035-3fdd-4f9c-a800-40802ac6839d","Type":"ContainerDied","Data":"c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92"}
Jul 13 21:49:46.648330 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.648317    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt" event={"ID":"32add035-3fdd-4f9c-a800-40802ac6839d","Type":"ContainerDied","Data":"bf8f0a8fdd110a7155ab4fd837bc4732b3ad36c391eea78d52753163960119c2"}
Jul 13 21:49:46.648474 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.648349    2564 scope.go:117] "RemoveContainer" containerID="c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92"
Jul 13 21:49:46.650388 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.650344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf" event={"ID":"c2aae361-16ab-4b3d-a1d9-6902e56a4570","Type":"ContainerStarted","Data":"5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704"}
Jul 13 21:49:46.650489 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.650433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf" event={"ID":"c2aae361-16ab-4b3d-a1d9-6902e56a4570","Type":"ContainerStarted","Data":"84b95c18bc1e83516ffdde0ade0fa7d91d91e603e0e98ca3717912a8a940281a"}
Jul 13 21:49:46.650653 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.650629    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:46.658689 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.658670    2564 scope.go:117] "RemoveContainer" containerID="713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c"
Jul 13 21:49:46.667311 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.667292    2564 scope.go:117] "RemoveContainer" containerID="c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92"
Jul 13 21:49:46.667564 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:49:46.667543    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92\": container with ID starting with c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92 not found: ID does not exist" containerID="c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92"
Jul 13 21:49:46.667675 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.667575    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92"} err="failed to get container status \"c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92\": rpc error: code = NotFound desc = could not find container \"c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92\": container with ID starting with c69f6af104630a53960e9160b86ff01fb16acd1b199fd3427772864ffb193d92 not found: ID does not exist"
Jul 13 21:49:46.667675 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.667617    2564 scope.go:117] "RemoveContainer" containerID="713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c"
Jul 13 21:49:46.667860 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:49:46.667843    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c\": container with ID starting with 713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c not found: ID does not exist" containerID="713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c"
Jul 13 21:49:46.667915 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.667871    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c"} err="failed to get container status \"713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c\": rpc error: code = NotFound desc = could not find container \"713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c\": container with ID starting with 713730c013d10bbddd5b712c52070686e0ef9f083be1ace16dbe77fc053e741c not found: ID does not exist"
Jul 13 21:49:46.670409 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.670349    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf" podStartSLOduration=1.2845403850000001 podStartE2EDuration="1.67033513s" totalImagesPullingTime="385.794745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:49:46.668258638 +0000 UTC m=+2433.325915644" watchObservedRunningTime="2026-07-13 21:49:46.67033513 +0000 UTC m=+2433.327992135"
Jul 13 21:49:46.670740 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.670722    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:46.670824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.670745    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32add035-3fdd-4f9c-a800-40802ac6839d-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:46.670824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.670757    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:46.670824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.670773    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5bfd\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-kube-api-access-g5bfd\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:46.670824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.670787    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:46.670824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.670802    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32add035-3fdd-4f9c-a800-40802ac6839d-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:46.670824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.670815    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32add035-3fdd-4f9c-a800-40802ac6839d-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:46.670824 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.670827    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32add035-3fdd-4f9c-a800-40802ac6839d-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:49:46.685952 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.685904    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"]
Jul 13 21:49:46.689995 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:46.689973    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-006a57-openshift-default-66cf9b4575-772vt"]
Jul 13 21:49:47.823632 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:47.823577    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" path="/var/lib/kubelet/pods/32add035-3fdd-4f9c-a800-40802ac6839d/volumes"
Jul 13 21:49:49.999752 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:49.999718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:50.313320 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.313242    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:50.577383 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.577305    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"]
Jul 13 21:49:50.577688 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.577668    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy"
Jul 13 21:49:50.577767 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.577692    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy"
Jul 13 21:49:50.577832 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.577819    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy"
Jul 13 21:49:50.577869 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.577834    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy"
Jul 13 21:49:50.580869 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.580849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.583314 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.583293    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-884921-openshift-default-dockercfg-lvkkb\""
Jul 13 21:49:50.591338 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.591312    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"]
Jul 13 21:49:50.599679 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.599654    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-podinfo\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.599774 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.599693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49cql\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-kube-api-access-49cql\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.599774 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.599727    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-socket\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.599848 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.599792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-data\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.599848 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.599832    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-credential-socket\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.599912 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.599881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-certs\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.599912 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.599901    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-token\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.599980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.599928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-envoy\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.599980 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.599972    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istiod-ca-cert\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.602991 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.602967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:50.701139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-socket\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701147    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-data\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-credential-socket\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-certs\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701243    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-token\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701285    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-envoy\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701334 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istiod-ca-cert\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701661 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-podinfo\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701661 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49cql\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-kube-api-access-49cql\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701661 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701578    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-socket\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701829 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701736    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-data\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.701912 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.701887    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-credential-socket\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.702327 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.702250    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-certs\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.702560 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.702418    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istiod-ca-cert\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.710736 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.710709    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-envoy\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.710961 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.710940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-podinfo\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.713396 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.713373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-token\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.713499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.713469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49cql\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-kube-api-access-49cql\") pod \"e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") " pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:50.716394 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.716367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:50.891655 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:50.891570    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:51.032179 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:51.032151    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"]
Jul 13 21:49:51.033633 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:49:51.033601    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72215c2a_97f3_42a8_8df6_fb3cd57bb6f9.slice/crio-1a6fa4bf2590ead68caec5990a854b85f11a1d39f632cb57f82d2d371f2cdc32 WatchSource:0}: Error finding container 1a6fa4bf2590ead68caec5990a854b85f11a1d39f632cb57f82d2d371f2cdc32: Status 404 returned error can't find the container with id 1a6fa4bf2590ead68caec5990a854b85f11a1d39f632cb57f82d2d371f2cdc32
Jul 13 21:49:51.035942 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:51.035906    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:49:51.036025 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:51.035971    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:49:51.036025 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:51.036005    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:49:51.679387 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:51.679347    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" event={"ID":"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9","Type":"ContainerStarted","Data":"5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5"}
Jul 13 21:49:51.679607 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:51.679400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" event={"ID":"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9","Type":"ContainerStarted","Data":"1a6fa4bf2590ead68caec5990a854b85f11a1d39f632cb57f82d2d371f2cdc32"}
Jul 13 21:49:51.701107 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:51.701062    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" podStartSLOduration=1.7010484830000001 podStartE2EDuration="1.701048483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:49:51.69891476 +0000 UTC m=+2438.356571792" watchObservedRunningTime="2026-07-13 21:49:51.701048483 +0000 UTC m=+2438.358705488"
Jul 13 21:49:51.892046 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:51.892012    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:52.668924 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:52.668888    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:49:52.897569 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:52.897537    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:53.562794 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.562738    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:53.661609 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.659196    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:53.673807 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.671152    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"]
Jul 13 21:49:53.673807 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.671747    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy"
Jul 13 21:49:53.673807 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.671768    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="32add035-3fdd-4f9c-a800-40802ac6839d" containerName="istio-proxy"
Jul 13 21:49:53.675405 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.675382    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:53.678436 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.678172    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"]
Jul 13 21:49:53.678560 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.678467    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-884921-serving-cert\""
Jul 13 21:49:53.689028 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.689003    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:53.690005 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.689986    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:49:53.729163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.729136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2hhr\" (UniqueName: \"kubernetes.io/projected/cb81f7c6-e31c-4109-8b50-11b59a7d9308-kube-api-access-r2hhr\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:53.729319 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.729195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb81f7c6-e31c-4109-8b50-11b59a7d9308-maas-api-tls\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:53.729319 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.729239    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb81f7c6-e31c-4109-8b50-11b59a7d9308-openshift-service-ca\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:53.736136 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.736095    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:53.830281 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.830202    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb81f7c6-e31c-4109-8b50-11b59a7d9308-maas-api-tls\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:53.830281 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.830251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb81f7c6-e31c-4109-8b50-11b59a7d9308-openshift-service-ca\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:53.830527 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.830306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2hhr\" (UniqueName: \"kubernetes.io/projected/cb81f7c6-e31c-4109-8b50-11b59a7d9308-kube-api-access-r2hhr\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:53.830527 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:49:53.830366    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-serving-cert: secret "maas-api-e2e-shared-b-884921-serving-cert" not found
Jul 13 21:49:53.830527 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:49:53.830452    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb81f7c6-e31c-4109-8b50-11b59a7d9308-maas-api-tls podName:cb81f7c6-e31c-4109-8b50-11b59a7d9308 nodeName:}" failed. No retries permitted until 2026-07-13 21:49:54.330432804 +0000 UTC m=+2440.988089795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb81f7c6-e31c-4109-8b50-11b59a7d9308-maas-api-tls") pod "maas-api-e2e-shared-b-884921-76785f54c7-lmjhg" (UID: "cb81f7c6-e31c-4109-8b50-11b59a7d9308") : secret "maas-api-e2e-shared-b-884921-serving-cert" not found
Jul 13 21:49:53.831080 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.831056    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb81f7c6-e31c-4109-8b50-11b59a7d9308-openshift-service-ca\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:53.841262 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.841239    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2hhr\" (UniqueName: \"kubernetes.io/projected/cb81f7c6-e31c-4109-8b50-11b59a7d9308-kube-api-access-r2hhr\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:53.895571 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:53.895530    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:54.335055 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:54.335025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb81f7c6-e31c-4109-8b50-11b59a7d9308-maas-api-tls\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:54.337782 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:54.337753    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb81f7c6-e31c-4109-8b50-11b59a7d9308-maas-api-tls\") pod \"maas-api-e2e-shared-b-884921-76785f54c7-lmjhg\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:54.343412 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:54.343310    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:54.448238 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:54.448153    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:54.588853 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:54.588769    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:54.768649 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:54.768099    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"]
Jul 13 21:49:55.700971 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:55.700930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg" event={"ID":"cb81f7c6-e31c-4109-8b50-11b59a7d9308","Type":"ContainerStarted","Data":"5d2028c221b244e2e4f5507f936d02971392e5d577b59a281cf4942aa23396f9"}
Jul 13 21:49:55.701192 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:55.700988    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg" event={"ID":"cb81f7c6-e31c-4109-8b50-11b59a7d9308","Type":"ContainerStarted","Data":"40021351a4508ee3018ae87b13ac0ca9597bfe45239f4cf6c7b28a6516d4eb5f"}
Jul 13 21:49:55.701269 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:55.701251    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:49:55.721252 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:55.721209    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg" podStartSLOduration=2.314527947 podStartE2EDuration="2.721195955s" totalImagesPullingTime="406.668008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:49:55.719063978 +0000 UTC m=+2442.376720985" watchObservedRunningTime="2026-07-13 21:49:55.721195955 +0000 UTC m=+2442.378852960"
Jul 13 21:49:58.021055 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:58.021023    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:49:58.717289 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:58.717262    2564 generic.go:356] "Generic (PLEG): container finished" podID="53133c74-3849-4ccb-8119-b2af7357b402" containerID="a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b" exitCode=0
Jul 13 21:49:58.717415 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:58.717330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" event={"ID":"53133c74-3849-4ccb-8119-b2af7357b402","Type":"ContainerDied","Data":"a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b"}
Jul 13 21:49:58.717746 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:58.717732    2564 scope.go:117] "RemoveContainer" containerID="a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b"
Jul 13 21:49:58.718686 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:58.718660    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:49:58.718753 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:58.718730    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:49:58.718791 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:58.718760    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:49:59.638249 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:59.638219    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:49:59.724093 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:49:59.724061    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" event={"ID":"53133c74-3849-4ccb-8119-b2af7357b402","Type":"ContainerStarted","Data":"6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81"}
Jul 13 21:50:00.451331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:00.450849    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:50:00.451331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:00.450981    2564 patch_prober.go:28] interesting pod/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 13 21:50:00.451331 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:00.451026    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 13 21:50:01.449743 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:01.449701    2564 patch_prober.go:28] interesting pod/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 13 21:50:01.450265 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:01.449767    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 13 21:50:01.713442 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:01.713369    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:50:02.450384 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:02.450345    2564 patch_prober.go:28] interesting pod/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 13 21:50:02.450929 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:02.450418    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 13 21:50:03.335137 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:03.335102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:50:03.450108 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:03.449646    2564 patch_prober.go:28] interesting pod/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 13 21:50:03.450108 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:03.449727    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 13 21:50:04.453812 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:04.453790    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:50:04.454217 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:04.454050    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:50:04.454846 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:04.454820    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:50:04.752388 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:04.752357    2564 generic.go:356] "Generic (PLEG): container finished" podID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerID="5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5" exitCode=0
Jul 13 21:50:04.752577 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:04.752438    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" event={"ID":"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9","Type":"ContainerDied","Data":"5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5"}
Jul 13 21:50:04.752987 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:04.752970    2564 scope.go:117] "RemoveContainer" containerID="5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5"
Jul 13 21:50:04.753857 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:04.753834    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:50:04.753941 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:04.753896    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:50:04.753941 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:04.753931    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"}
Jul 13 21:50:05.763001 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:05.762963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" event={"ID":"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9","Type":"ContainerStarted","Data":"b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07"}
Jul 13 21:50:05.892569 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:05.892530    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:50:05.893896 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:05.893871    2564 patch_prober.go:28] interesting pod/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 13 21:50:05.894025 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:05.893918    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 13 21:50:06.892096 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:06.892057    2564 patch_prober.go:28] interesting pod/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 13 21:50:06.892481 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:06.892117    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 13 21:50:06.979269 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:06.979233    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:50:07.074380 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:07.074342    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:50:07.892176 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:07.892136    2564 patch_prober.go:28] interesting pod/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 13 21:50:07.892552 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:07.892189    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 13 21:50:08.688983 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:08.688952    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:50:08.891972 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:08.891942    2564 patch_prober.go:28] interesting pod/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 13 21:50:08.892139 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:08.891991    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 13 21:50:09.895952 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:09.895925    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:50:09.896961 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:09.896939    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:50:15.289810 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:15.289778    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:50:15.392102 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:15.392073    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:50:32.384655 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:50:32.384619    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:52:36.990813 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:52:36.990719    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:52:38.195000 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:52:38.194962    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:52:42.703110 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:52:42.703074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:52:47.731720 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:52:47.731687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:52:47.835407 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:52:47.835368    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:52:51.405892 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:52:51.405861    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:52:51.457726 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:52:51.457691    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:53:07.889799 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:53:07.889764    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:53:42.918876 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:53:42.918843    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:53:43.859287 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:53:43.859254    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:53:46.412303 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:53:46.412260    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:54:13.975319 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:54:13.975246    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:54:13.983159 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:54:13.983139    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx6cw_ff20edcf-5053-44fb-af08-bc34f7a74a49/ovn-acl-logging/0.log"
Jul 13 21:54:32.004549 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:54:32.004514    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:07.827744 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:07.827711    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:07.932483 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:07.932447    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:16.016722 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:16.016684    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:16.113881 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:16.113846    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:20.296802 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:20.296763    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:23.466810 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:23.466771    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:24.367691 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:24.367659    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:44.407367 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:44.407273    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:44.510806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:44.510777    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:54.310918 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.310871    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"]
Jul 13 21:55:54.311490 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.311126    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf" podUID="c2aae361-16ab-4b3d-a1d9-6902e56a4570" containerName="maas-api" containerID="cri-o://5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704" gracePeriod=30
Jul 13 21:55:54.337992 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.337967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:55:54.564038 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.563977    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:55:54.660720 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.660692    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc7lz\" (UniqueName: \"kubernetes.io/projected/c2aae361-16ab-4b3d-a1d9-6902e56a4570-kube-api-access-dc7lz\") pod \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") "
Jul 13 21:55:54.660866 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.660731    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aae361-16ab-4b3d-a1d9-6902e56a4570-openshift-service-ca\") pod \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") "
Jul 13 21:55:54.660866 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.660753    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aae361-16ab-4b3d-a1d9-6902e56a4570-maas-api-tls\") pod \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\" (UID: \"c2aae361-16ab-4b3d-a1d9-6902e56a4570\") "
Jul 13 21:55:54.661088 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.661065    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2aae361-16ab-4b3d-a1d9-6902e56a4570-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2aae361-16ab-4b3d-a1d9-6902e56a4570" (UID: "c2aae361-16ab-4b3d-a1d9-6902e56a4570"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:55:54.662914 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.662889    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2aae361-16ab-4b3d-a1d9-6902e56a4570-kube-api-access-dc7lz" (OuterVolumeSpecName: "kube-api-access-dc7lz") pod "c2aae361-16ab-4b3d-a1d9-6902e56a4570" (UID: "c2aae361-16ab-4b3d-a1d9-6902e56a4570"). InnerVolumeSpecName "kube-api-access-dc7lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:55:54.663019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.662942    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2aae361-16ab-4b3d-a1d9-6902e56a4570-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2aae361-16ab-4b3d-a1d9-6902e56a4570" (UID: "c2aae361-16ab-4b3d-a1d9-6902e56a4570"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:55:54.761797 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.761772    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc7lz\" (UniqueName: \"kubernetes.io/projected/c2aae361-16ab-4b3d-a1d9-6902e56a4570-kube-api-access-dc7lz\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:55:54.761797 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.761796    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aae361-16ab-4b3d-a1d9-6902e56a4570-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:55:54.761943 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:54.761807    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aae361-16ab-4b3d-a1d9-6902e56a4570-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:55:55.429479 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.429445    2564 generic.go:356] "Generic (PLEG): container finished" podID="c2aae361-16ab-4b3d-a1d9-6902e56a4570" containerID="5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704" exitCode=0
Jul 13 21:55:55.430019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.429503    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"
Jul 13 21:55:55.430019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.429514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf" event={"ID":"c2aae361-16ab-4b3d-a1d9-6902e56a4570","Type":"ContainerDied","Data":"5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704"}
Jul 13 21:55:55.430019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.429557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf" event={"ID":"c2aae361-16ab-4b3d-a1d9-6902e56a4570","Type":"ContainerDied","Data":"84b95c18bc1e83516ffdde0ade0fa7d91d91e603e0e98ca3717912a8a940281a"}
Jul 13 21:55:55.430019 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.429597    2564 scope.go:117] "RemoveContainer" containerID="5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704"
Jul 13 21:55:55.438822 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.438806    2564 scope.go:117] "RemoveContainer" containerID="5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704"
Jul 13 21:55:55.439056 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:55:55.439037    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704\": container with ID starting with 5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704 not found: ID does not exist" containerID="5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704"
Jul 13 21:55:55.439102 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.439066    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704"} err="failed to get container status \"5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704\": rpc error: code = NotFound desc = could not find container \"5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704\": container with ID starting with 5e5793e50d270173c0e16a64f0beabe8eb0d3582205227c3db53928827efa704 not found: ID does not exist"
Jul 13 21:55:55.450648 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.450625    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"]
Jul 13 21:55:55.453621 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.453600    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e2974-76c8c96d9c-ntvrf"]
Jul 13 21:55:55.821901 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:55.821873    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2aae361-16ab-4b3d-a1d9-6902e56a4570" path="/var/lib/kubelet/pods/c2aae361-16ab-4b3d-a1d9-6902e56a4570/volumes"
Jul 13 21:55:59.008870 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:55:59.008839    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:04.897347 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:04.897316    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:04.957124 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:04.957088    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:09.639714 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:09.639686    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"]
Jul 13 21:56:09.640100 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:09.639901    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy" containerID="cri-o://6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81" gracePeriod=30
Jul 13 21:56:09.659739 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:09.659710    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:09.896486 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:09.896390    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:10.572954 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:10.572920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:10.672860 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:10.672832    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:14.884012 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:14.883989    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:56:15.018965 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.018938    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-socket\") pod \"53133c74-3849-4ccb-8119-b2af7357b402\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") "
Jul 13 21:56:15.019117 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.018971    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-certs\") pod \"53133c74-3849-4ccb-8119-b2af7357b402\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") "
Jul 13 21:56:15.019117 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.018991    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-envoy\") pod \"53133c74-3849-4ccb-8119-b2af7357b402\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") "
Jul 13 21:56:15.019117 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019025    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-credential-socket\") pod \"53133c74-3849-4ccb-8119-b2af7357b402\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") "
Jul 13 21:56:15.019117 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019076    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-data\") pod \"53133c74-3849-4ccb-8119-b2af7357b402\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") "
Jul 13 21:56:15.019117 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019104    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-istio-token\") pod \"53133c74-3849-4ccb-8119-b2af7357b402\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") "
Jul 13 21:56:15.019374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019151    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53133c74-3849-4ccb-8119-b2af7357b402-istiod-ca-cert\") pod \"53133c74-3849-4ccb-8119-b2af7357b402\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") "
Jul 13 21:56:15.019374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019171    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53133c74-3849-4ccb-8119-b2af7357b402-istio-podinfo\") pod \"53133c74-3849-4ccb-8119-b2af7357b402\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") "
Jul 13 21:56:15.019374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019207    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jnm7\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-kube-api-access-6jnm7\") pod \"53133c74-3849-4ccb-8119-b2af7357b402\" (UID: \"53133c74-3849-4ccb-8119-b2af7357b402\") "
Jul 13 21:56:15.019374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019246    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "53133c74-3849-4ccb-8119-b2af7357b402" (UID: "53133c74-3849-4ccb-8119-b2af7357b402"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:15.019374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019259    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "53133c74-3849-4ccb-8119-b2af7357b402" (UID: "53133c74-3849-4ccb-8119-b2af7357b402"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:15.019635 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019460    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "53133c74-3849-4ccb-8119-b2af7357b402" (UID: "53133c74-3849-4ccb-8119-b2af7357b402"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:15.019635 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019506    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:15.019635 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019526    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:15.019635 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.019576    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53133c74-3849-4ccb-8119-b2af7357b402-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "53133c74-3849-4ccb-8119-b2af7357b402" (UID: "53133c74-3849-4ccb-8119-b2af7357b402"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:56:15.020332 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.020289    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-data" (OuterVolumeSpecName: "istio-data") pod "53133c74-3849-4ccb-8119-b2af7357b402" (UID: "53133c74-3849-4ccb-8119-b2af7357b402"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:15.021481 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.021450    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "53133c74-3849-4ccb-8119-b2af7357b402" (UID: "53133c74-3849-4ccb-8119-b2af7357b402"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:15.021576 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.021526    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-istio-token" (OuterVolumeSpecName: "istio-token") pod "53133c74-3849-4ccb-8119-b2af7357b402" (UID: "53133c74-3849-4ccb-8119-b2af7357b402"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:56:15.021732 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.021706    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53133c74-3849-4ccb-8119-b2af7357b402-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "53133c74-3849-4ccb-8119-b2af7357b402" (UID: "53133c74-3849-4ccb-8119-b2af7357b402"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:56:15.021846 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.021828    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-kube-api-access-6jnm7" (OuterVolumeSpecName: "kube-api-access-6jnm7") pod "53133c74-3849-4ccb-8119-b2af7357b402" (UID: "53133c74-3849-4ccb-8119-b2af7357b402"). InnerVolumeSpecName "kube-api-access-6jnm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:56:15.120038 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.120015    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:15.120038 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.120037    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:15.120168 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.120046    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53133c74-3849-4ccb-8119-b2af7357b402-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:15.120168 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.120054    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53133c74-3849-4ccb-8119-b2af7357b402-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:15.120168 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.120062    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jnm7\" (UniqueName: \"kubernetes.io/projected/53133c74-3849-4ccb-8119-b2af7357b402-kube-api-access-6jnm7\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:15.120168 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.120072    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:15.120168 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.120080    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53133c74-3849-4ccb-8119-b2af7357b402-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:15.529101 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.529068    2564 generic.go:356] "Generic (PLEG): container finished" podID="53133c74-3849-4ccb-8119-b2af7357b402" containerID="6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81" exitCode=0
Jul 13 21:56:15.529252 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.529121    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"
Jul 13 21:56:15.529252 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.529150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" event={"ID":"53133c74-3849-4ccb-8119-b2af7357b402","Type":"ContainerDied","Data":"6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81"}
Jul 13 21:56:15.529252 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.529197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq" event={"ID":"53133c74-3849-4ccb-8119-b2af7357b402","Type":"ContainerDied","Data":"e608df0ed1a32d984795661180a5beea9620343ff4136ff78be178ec3b12c24e"}
Jul 13 21:56:15.529252 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.529218    2564 scope.go:117] "RemoveContainer" containerID="6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81"
Jul 13 21:56:15.538118 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.538096    2564 scope.go:117] "RemoveContainer" containerID="a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b"
Jul 13 21:56:15.546402 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.546386    2564 scope.go:117] "RemoveContainer" containerID="6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81"
Jul 13 21:56:15.546653 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:56:15.546635    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81\": container with ID starting with 6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81 not found: ID does not exist" containerID="6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81"
Jul 13 21:56:15.546705 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.546662    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81"} err="failed to get container status \"6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81\": rpc error: code = NotFound desc = could not find container \"6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81\": container with ID starting with 6c7c1ee67ec1dcf52eb36aa18dcf221a06a3d7f58c77854a96f75cd2fb134c81 not found: ID does not exist"
Jul 13 21:56:15.546705 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.546680    2564 scope.go:117] "RemoveContainer" containerID="a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b"
Jul 13 21:56:15.546953 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:56:15.546936    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b\": container with ID starting with a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b not found: ID does not exist" containerID="a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b"
Jul 13 21:56:15.546994 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.546961    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b"} err="failed to get container status \"a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b\": rpc error: code = NotFound desc = could not find container \"a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b\": container with ID starting with a5e4b8c773fac23580241b3bfd67539161d7737620c77e93fcc50889b4fe672b not found: ID does not exist"
Jul 13 21:56:15.550726 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.550703    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"]
Jul 13 21:56:15.554700 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.554681    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-8e2974-openshift-default-56ff4d547-vqdvq"]
Jul 13 21:56:15.821784 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:15.821714    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53133c74-3849-4ccb-8119-b2af7357b402" path="/var/lib/kubelet/pods/53133c74-3849-4ccb-8119-b2af7357b402/volumes"
Jul 13 21:56:20.378916 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.378880    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"]
Jul 13 21:56:20.379325 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.379109    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg" podUID="cb81f7c6-e31c-4109-8b50-11b59a7d9308" containerName="maas-api" containerID="cri-o://5d2028c221b244e2e4f5507f936d02971392e5d577b59a281cf4942aa23396f9" gracePeriod=30
Jul 13 21:56:20.404326 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.404301    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:20.556338 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.556310    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb81f7c6-e31c-4109-8b50-11b59a7d9308" containerID="5d2028c221b244e2e4f5507f936d02971392e5d577b59a281cf4942aa23396f9" exitCode=0
Jul 13 21:56:20.556495 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.556404    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg" event={"ID":"cb81f7c6-e31c-4109-8b50-11b59a7d9308","Type":"ContainerDied","Data":"5d2028c221b244e2e4f5507f936d02971392e5d577b59a281cf4942aa23396f9"}
Jul 13 21:56:20.628560 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.628537    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:56:20.763993 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.763966    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb81f7c6-e31c-4109-8b50-11b59a7d9308-maas-api-tls\") pod \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") "
Jul 13 21:56:20.764159 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.764007    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb81f7c6-e31c-4109-8b50-11b59a7d9308-openshift-service-ca\") pod \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") "
Jul 13 21:56:20.764159 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.764061    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2hhr\" (UniqueName: \"kubernetes.io/projected/cb81f7c6-e31c-4109-8b50-11b59a7d9308-kube-api-access-r2hhr\") pod \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\" (UID: \"cb81f7c6-e31c-4109-8b50-11b59a7d9308\") "
Jul 13 21:56:20.764353 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.764332    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb81f7c6-e31c-4109-8b50-11b59a7d9308-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb81f7c6-e31c-4109-8b50-11b59a7d9308" (UID: "cb81f7c6-e31c-4109-8b50-11b59a7d9308"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:56:20.766137 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.766116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb81f7c6-e31c-4109-8b50-11b59a7d9308-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb81f7c6-e31c-4109-8b50-11b59a7d9308" (UID: "cb81f7c6-e31c-4109-8b50-11b59a7d9308"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:56:20.766258 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.766237    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb81f7c6-e31c-4109-8b50-11b59a7d9308-kube-api-access-r2hhr" (OuterVolumeSpecName: "kube-api-access-r2hhr") pod "cb81f7c6-e31c-4109-8b50-11b59a7d9308" (UID: "cb81f7c6-e31c-4109-8b50-11b59a7d9308"). InnerVolumeSpecName "kube-api-access-r2hhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:56:20.865675 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.865649    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb81f7c6-e31c-4109-8b50-11b59a7d9308-maas-api-tls\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:20.865675 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.865675    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb81f7c6-e31c-4109-8b50-11b59a7d9308-openshift-service-ca\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:20.865813 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:20.865685    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2hhr\" (UniqueName: \"kubernetes.io/projected/cb81f7c6-e31c-4109-8b50-11b59a7d9308-kube-api-access-r2hhr\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:21.561848 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:21.561819    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"
Jul 13 21:56:21.561848 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:21.561844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg" event={"ID":"cb81f7c6-e31c-4109-8b50-11b59a7d9308","Type":"ContainerDied","Data":"40021351a4508ee3018ae87b13ac0ca9597bfe45239f4cf6c7b28a6516d4eb5f"}
Jul 13 21:56:21.562362 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:21.561898    2564 scope.go:117] "RemoveContainer" containerID="5d2028c221b244e2e4f5507f936d02971392e5d577b59a281cf4942aa23396f9"
Jul 13 21:56:21.583786 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:21.583762    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"]
Jul 13 21:56:21.588096 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:21.588071    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-884921-76785f54c7-lmjhg"]
Jul 13 21:56:21.821854 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:21.821784    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb81f7c6-e31c-4109-8b50-11b59a7d9308" path="/var/lib/kubelet/pods/cb81f7c6-e31c-4109-8b50-11b59a7d9308/volumes"
Jul 13 21:56:25.093531 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:25.093461    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:30.875095 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:30.875056    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:30.932794 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:30.932769    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:35.181826 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:35.181785    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:35.281296 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:35.281254    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:35.724820 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:35.724786    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"]
Jul 13 21:56:35.725044 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:35.725020    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy" containerID="cri-o://b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07" gracePeriod=30
Jul 13 21:56:35.771913 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:35.771877    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:35.955323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:35.955288    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:39.898215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:39.898177    2564 patch_prober.go:28] interesting pod/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 21:56:39.898669 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:39.898252    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:56:40.975360 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:40.975340    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:56:41.113084 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113056    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-podinfo\") pod \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") "
Jul 13 21:56:41.113215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113099    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-credential-socket\") pod \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") "
Jul 13 21:56:41.113215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113117    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-certs\") pod \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") "
Jul 13 21:56:41.113215 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113136    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-socket\") pod \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") "
Jul 13 21:56:41.113382 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113225    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-token\") pod \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") "
Jul 13 21:56:41.113382 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113259    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-envoy\") pod \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") "
Jul 13 21:56:41.113382 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113287    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-data\") pod \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") "
Jul 13 21:56:41.113382 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113319    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49cql\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-kube-api-access-49cql\") pod \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") "
Jul 13 21:56:41.113620 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113357    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" (UID: "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:41.113620 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113405    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" (UID: "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:41.113620 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" (UID: "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:41.113620 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113617    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istiod-ca-cert\") pod \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\" (UID: \"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9\") "
Jul 13 21:56:41.113869 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113851    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-credential-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:41.113935 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113873    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-certs\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:41.113935 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113911    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-workload-socket\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:41.113935 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.113917    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" (UID: "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:56:41.114231 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.114211    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" (UID: "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:41.116198 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.116173    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" (UID: "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 21:56:41.116198 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.116185    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-kube-api-access-49cql" (OuterVolumeSpecName: "kube-api-access-49cql") pod "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" (UID: "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9"). InnerVolumeSpecName "kube-api-access-49cql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:56:41.116322 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.116225    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" (UID: "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:56:41.116322 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.116239    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" (UID: "72215c2a-97f3-42a8-8df6-fb3cd57bb6f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:56:41.205221 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.205198    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66ffdf856b-zh9sw_b4c13ca7-df5e-452f-829e-2575b90a05e1/manager/0.log"
Jul 13 21:56:41.215109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.215091    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-token\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:41.215109 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.215109    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-envoy\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:41.215218 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.215117    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-data\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:41.215218 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.215125    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49cql\" (UniqueName: \"kubernetes.io/projected/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-kube-api-access-49cql\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:41.215218 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.215133    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istiod-ca-cert\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:41.215218 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.215143    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9-istio-podinfo\") on node \"ip-10-0-141-152.ec2.internal\" DevicePath \"\""
Jul 13 21:56:41.652420 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.652395    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fb46456c-qwbmx_bd4b197e-9847-4de7-b873-75a5bb23822a/copy-manifests/0.log"
Jul 13 21:56:41.659939 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.659903    2564 generic.go:356] "Generic (PLEG): container finished" podID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerID="b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07" exitCode=0
Jul 13 21:56:41.660084 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.659964    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"
Jul 13 21:56:41.660084 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.660024    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" event={"ID":"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9","Type":"ContainerDied","Data":"b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07"}
Jul 13 21:56:41.660212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.660088    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz" event={"ID":"72215c2a-97f3-42a8-8df6-fb3cd57bb6f9","Type":"ContainerDied","Data":"1a6fa4bf2590ead68caec5990a854b85f11a1d39f632cb57f82d2d371f2cdc32"}
Jul 13 21:56:41.660212 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.660124    2564 scope.go:117] "RemoveContainer" containerID="b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07"
Jul 13 21:56:41.672385 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.672361    2564 scope.go:117] "RemoveContainer" containerID="5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5"
Jul 13 21:56:41.673047 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.673029    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fb46456c-qwbmx_bd4b197e-9847-4de7-b873-75a5bb23822a/manager/0.log"
Jul 13 21:56:41.680323 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.680306    2564 scope.go:117] "RemoveContainer" containerID="b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07"
Jul 13 21:56:41.680560 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:56:41.680541    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07\": container with ID starting with b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07 not found: ID does not exist" containerID="b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07"
Jul 13 21:56:41.680660 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.680573    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07"} err="failed to get container status \"b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07\": rpc error: code = NotFound desc = could not find container \"b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07\": container with ID starting with b7718b7da84af5af2e83ddacc117c0468b80ef70425dc5dd3d04ed1df7bc9b07 not found: ID does not exist"
Jul 13 21:56:41.680660 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.680612    2564 scope.go:117] "RemoveContainer" containerID="5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5"
Jul 13 21:56:41.680879 ip-10-0-141-152 kubenswrapper[2564]: E0713 21:56:41.680862    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5\": container with ID starting with 5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5 not found: ID does not exist" containerID="5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5"
Jul 13 21:56:41.680922 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.680885    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5"} err="failed to get container status \"5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5\": rpc error: code = NotFound desc = could not find container \"5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5\": container with ID starting with 5c16eaac65660c67968a3682b536dd3c1585d780e147e1147291ba2838082bd5 not found: ID does not exist"
Jul 13 21:56:41.686066 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.686045    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"]
Jul 13 21:56:41.689677 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.689642    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-884921-openshift-default-7c45486b7f-8p4tz"]
Jul 13 21:56:41.821976 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:41.821953    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" path="/var/lib/kubelet/pods/72215c2a-97f3-42a8-8df6-fb3cd57bb6f9/volumes"
Jul 13 21:56:42.137338 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:42.137310    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5dcfb87784-2t785_0cf8cc9a-88eb-40b8-9dee-9f303dbacd8b/maas-api/0.log"
Jul 13 21:56:42.244002 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:42.243982    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732970-dqbfc_7bc7b126-cb9b-4218-8450-eea9e3cdf6a5/cleanup/0.log"
Jul 13 21:56:42.351944 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:42.351928    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732985-qlkmq_86f68d2c-5bbf-4516-bab4-8b4781e055c3/cleanup/0.log"
Jul 13 21:56:42.463722 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:42.463659    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-057af81a-99cqh_1e9fd6c9-6901-417e-bcd4-30362aa72eef/revoke-keys/0.log"
Jul 13 21:56:42.580534 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:42.580512    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-6f1f77b0-xglm7_f7421761-f854-423e-a1c9-04796ac4821e/revoke-keys/0.log"
Jul 13 21:56:42.915184 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:42.915155    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-ae8a7501-74sfz_ee0eb5b7-2c49-4df7-ae24-9808bc5498f8/revoke-keys/0.log"
Jul 13 21:56:43.241853 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:43.241829    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-f4738480-smsmh_169672e1-1952-4ec7-9b48-aa62bd679ab6/revoke-keys/0.log"
Jul 13 21:56:43.363338 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:43.363310    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-6c7d8be7-22vgn_da7cf9ff-3f9a-4a25-ac27-615356e24fba/revoke-keys/0.log"
Jul 13 21:56:43.474543 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:43.474508    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-8603bf9d-cw9zx_10b802c1-0996-4c3a-92b1-03c96f602cb7/revoke-keys/0.log"
Jul 13 21:56:46.247067 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:46.247033    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mz4h4_dba4abee-c858-416f-90d1-93430f81c664/discovery/0.log"
Jul 13 21:56:46.696818 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:46.696728    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-h48dj_b2a7afd7-7a18-4b08-85ac-7787ef89fa11/payload-pre-processing/0.log"
Jul 13 21:56:47.344532 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:47.344498    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-svbnb_776c9e66-8c56-4622-91db-9c06ae13662f/main/0.log"
Jul 13 21:56:47.553499 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:47.553465    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfp9mw_ecbc313a-2b4f-427b-aa44-79623dd0de68/main/0.log"
Jul 13 21:56:51.243683 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:51.243643    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:51.342776 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:51.342739    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:53.745200 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:53.745164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:56:54.044136 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:54.044053    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xl4b6_166204bf-e576-42fa-80df-02d81b311427/global-pull-secret-syncer/0.log"
Jul 13 21:56:54.097423 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:54.097391    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log"
Jul 13 21:56:54.242228 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:56:54.242201    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-152.ec2.internal_05a87461878d507ac253066a80c27967/haproxy/0.log"
Jul 13 21:57:02.434951 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:02.434920    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-grslf_2afea579-811d-46ea-8cd9-b7561d66efcb/node-exporter/0.log"
Jul 13 21:57:02.470773 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:02.470748    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-grslf_2afea579-811d-46ea-8cd9-b7561d66efcb/kube-rbac-proxy/0.log"
Jul 13 21:57:02.512343 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:02.512325    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-grslf_2afea579-811d-46ea-8cd9-b7561d66efcb/init-textfile/0.log"
Jul 13 21:57:04.172424 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172392    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"]
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172689    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172700    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172715    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172720    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172728    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172734    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172743    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2aae361-16ab-4b3d-a1d9-6902e56a4570" containerName="maas-api"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172748    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2aae361-16ab-4b3d-a1d9-6902e56a4570" containerName="maas-api"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172756    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172761    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172765    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb81f7c6-e31c-4109-8b50-11b59a7d9308" containerName="maas-api"
Jul 13 21:57:04.172806 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172770    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb81f7c6-e31c-4109-8b50-11b59a7d9308" containerName="maas-api"
Jul 13 21:57:04.173163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172816    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb81f7c6-e31c-4109-8b50-11b59a7d9308" containerName="maas-api"
Jul 13 21:57:04.173163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172823    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy"
Jul 13 21:57:04.173163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172829    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72215c2a-97f3-42a8-8df6-fb3cd57bb6f9" containerName="istio-proxy"
Jul 13 21:57:04.173163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172836    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="53133c74-3849-4ccb-8119-b2af7357b402" containerName="istio-proxy"
Jul 13 21:57:04.173163 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.172841    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2aae361-16ab-4b3d-a1d9-6902e56a4570" containerName="maas-api"
Jul 13 21:57:04.175507 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.175490    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.178071 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.178049    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s2kv7\"/\"openshift-service-ca.crt\""
Jul 13 21:57:04.178211 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.178095    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s2kv7\"/\"default-dockercfg-v2jnb\""
Jul 13 21:57:04.179173 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.179154    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s2kv7\"/\"kube-root-ca.crt\""
Jul 13 21:57:04.191393 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.191373    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"]
Jul 13 21:57:04.279070 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.279044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxv6\" (UniqueName: \"kubernetes.io/projected/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-kube-api-access-6dxv6\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.279191 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.279103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-proc\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.279191 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.279133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-sys\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.279191 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.279173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-lib-modules\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.279374 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.279215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-podres\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.380350 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.380329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-proc\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.380465 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.380358    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-sys\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.380465 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.380382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-lib-modules\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.380465 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.380400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-podres\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.380465 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.380427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxv6\" (UniqueName: \"kubernetes.io/projected/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-kube-api-access-6dxv6\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.380465 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.380458    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-proc\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.380718 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.380465    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-sys\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.380718 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.380520    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-lib-modules\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.380718 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.380521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-podres\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.388127 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.388108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxv6\" (UniqueName: \"kubernetes.io/projected/7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3-kube-api-access-6dxv6\") pod \"perf-node-gather-daemonset-tb28g\" (UID: \"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3\") " pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.460211 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.460154    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-wc5th_e278d43d-f638-4047-96ca-a5ef535138d7/networking-console-plugin/0.log"
Jul 13 21:57:04.485389 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.485368    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.618486 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.618464    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"]
Jul 13 21:57:04.621225 ip-10-0-141-152 kubenswrapper[2564]: W0713 21:57:04.621200    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7e5c572d_0cc2_43df_a3f6_1a02ba1d8fc3.slice/crio-962cf1596da5022faf313537ab52e63f334e896ce0d35b91b7ccc84f3aac03de WatchSource:0}: Error finding container 962cf1596da5022faf313537ab52e63f334e896ce0d35b91b7ccc84f3aac03de: Status 404 returned error can't find the container with id 962cf1596da5022faf313537ab52e63f334e896ce0d35b91b7ccc84f3aac03de
Jul 13 21:57:04.771871 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.771835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g" event={"ID":"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3","Type":"ContainerStarted","Data":"156796b18de02b548a605f0da9282d3ee44fe6a74ee486d1f4b4c840e1de3659"}
Jul 13 21:57:04.772010 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.771890    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g" event={"ID":"7e5c572d-0cc2-43df-a3f6-1a02ba1d8fc3","Type":"ContainerStarted","Data":"962cf1596da5022faf313537ab52e63f334e896ce0d35b91b7ccc84f3aac03de"}
Jul 13 21:57:04.772010 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.771954    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:04.788052 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:04.788009    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g" podStartSLOduration=0.787996453 podStartE2EDuration="787.996453ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:57:04.787079331 +0000 UTC m=+2871.444736337" watchObservedRunningTime="2026-07-13 21:57:04.787996453 +0000 UTC m=+2871.445653459"
Jul 13 21:57:07.132757 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:07.132698    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s2fq8_fd368277-a7bf-4c9e-bdcc-90aea0f41dbb/dns/0.log"
Jul 13 21:57:07.151899 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:07.151862    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s2fq8_fd368277-a7bf-4c9e-bdcc-90aea0f41dbb/kube-rbac-proxy/0.log"
Jul 13 21:57:07.266392 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:07.266368    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s4zzn_30ac7b1f-d387-4c5a-9b4a-3c99304f8d45/dns-node-resolver/0.log"
Jul 13 21:57:07.825925 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:07.825903    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcjdj_f76f5b2b-3323-4299-9eef-aa9bc1763402/node-ca/0.log"
Jul 13 21:57:08.965948 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:08.965916    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mz4h4_dba4abee-c858-416f-90d1-93430f81c664/discovery/0.log"
Jul 13 21:57:09.121328 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:09.121295    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-h48dj_b2a7afd7-7a18-4b08-85ac-7787ef89fa11/payload-pre-processing/0.log"
Jul 13 21:57:09.866696 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:09.866659    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m8nwq_a1e66718-f6ca-466b-80f6-ee99fc06444f/serve-healthcheck-canary/0.log"
Jul 13 21:57:10.384778 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:10.384751    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gbwwc_5f99a209-3a7e-47c8-b6b6-c7c498775028/kube-rbac-proxy/0.log"
Jul 13 21:57:10.407135 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:10.407105    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gbwwc_5f99a209-3a7e-47c8-b6b6-c7c498775028/exporter/0.log"
Jul 13 21:57:10.428383 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:10.428359    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gbwwc_5f99a209-3a7e-47c8-b6b6-c7c498775028/extractor/0.log"
Jul 13 21:57:10.785960 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:10.785938    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s2kv7/perf-node-gather-daemonset-tb28g"
Jul 13 21:57:13.007698 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:13.007650    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66ffdf856b-zh9sw_b4c13ca7-df5e-452f-829e-2575b90a05e1/manager/0.log"
Jul 13 21:57:13.388664 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:13.388637    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fb46456c-qwbmx_bd4b197e-9847-4de7-b873-75a5bb23822a/manager/0.log"
Jul 13 21:57:13.406682 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:13.406657    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fb46456c-qwbmx_bd4b197e-9847-4de7-b873-75a5bb23822a/copy-manifests/0.log"
Jul 13 21:57:14.851067 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:14.851037    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-588f9fd4c6-f957r_5fd0e308-c964-4d14-8c4a-f305ec1292e9/manager/0.log"
Jul 13 21:57:14.877964 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:14.877938    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vsdh2_d5850608-d7f8-4a3a-8d5a-d17d69c4ddfd/openshift-lws-operator/0.log"
Jul 13 21:57:15.306217 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:15.306190    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mz4h4_dba4abee-c858-416f-90d1-93430f81c664/discovery/0.log"
Jul 13 21:57:15.425605 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:15.425513    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-h48dj_b2a7afd7-7a18-4b08-85ac-7787ef89fa11/payload-pre-processing/0.log"
Jul 13 21:57:16.895835 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:16.895800    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mz4h4_dba4abee-c858-416f-90d1-93430f81c664/discovery/0.log"
Jul 13 21:57:17.016602 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:17.016550    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-h48dj_b2a7afd7-7a18-4b08-85ac-7787ef89fa11/payload-pre-processing/0.log"
Jul 13 21:57:20.575860 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:20.575816    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:20.633856 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:20.633815    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4sdl_99b9abac-1432-4aec-b1c7-3e2550422583/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:21.371629 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.371578    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwxvt_7173a77a-9b2f-4837-81d5-0863d79a9008/kube-multus-additional-cni-plugins/0.log"
Jul 13 21:57:21.391401 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.391376    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwxvt_7173a77a-9b2f-4837-81d5-0863d79a9008/egress-router-binary-copy/0.log"
Jul 13 21:57:21.411307 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.411285    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwxvt_7173a77a-9b2f-4837-81d5-0863d79a9008/cni-plugins/0.log"
Jul 13 21:57:21.430498 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.430480    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwxvt_7173a77a-9b2f-4837-81d5-0863d79a9008/bond-cni-plugin/0.log"
Jul 13 21:57:21.449628 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.449612    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwxvt_7173a77a-9b2f-4837-81d5-0863d79a9008/routeoverride-cni/0.log"
Jul 13 21:57:21.469220 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.469203    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwxvt_7173a77a-9b2f-4837-81d5-0863d79a9008/whereabouts-cni-bincopy/0.log"
Jul 13 21:57:21.489301 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.489275    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwxvt_7173a77a-9b2f-4837-81d5-0863d79a9008/whereabouts-cni/0.log"
Jul 13 21:57:21.537406 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.537382    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-j7r2w_a07bee79-0d6e-439f-b920-be4cd37930b3/kube-multus/0.log"
Jul 13 21:57:21.659052 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.658991    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fwj6w_6e598283-d5e4-489e-a446-3a4fca30fb56/network-metrics-daemon/0.log"
Jul 13 21:57:21.676414 ip-10-0-141-152 kubenswrapper[2564]: I0713 21:57:21.676391    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fwj6w_6e598283-d5e4-489e-a446-3a4fca30fb56/kube-rbac-proxy/0.log"