Jul 29 00:36:08.152735 ip-10-0-132-227 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 00:36:08.554928 ip-10-0-132-227 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 00:36:08.554928 ip-10-0-132-227 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 00:36:08.554928 ip-10-0-132-227 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 00:36:08.554928 ip-10-0-132-227 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 00:36:08.554928 ip-10-0-132-227 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 00:36:08.557239 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.557095    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 00:36:08.562799 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562783    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 00:36:08.562799 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562798    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562801    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562805    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562808    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562810    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562813    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562817    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562820    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562823    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562825    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562828    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562831    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562833    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562836    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562842    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562846    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562849    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562852    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562854    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 00:36:08.562864 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562857    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562859    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562862    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562865    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562867    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562870    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562873    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562875    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562878    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562881    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562883    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562886    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562889    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562892    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562894    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562897    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562899    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562902    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562904    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562907    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 00:36:08.563341 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562910    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562912    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562915    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562917    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562919    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562922    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562925    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562927    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562930    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562932    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562935    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562937    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562941    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562945    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562948    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562952    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562955    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562958    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562961    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 00:36:08.563844 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562964    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562967    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562970    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562973    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562976    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562978    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562981    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562983    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562986    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562988    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562990    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562993    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562996    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.562998    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563001    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563004    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563007    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563009    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563011    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563015    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 00:36:08.564295 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563018    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563020    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563022    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563025    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563027    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563030    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563033    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563037    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563039    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563042    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563045    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563047    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563049    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563052    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563054    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563057    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563059    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563062    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563064    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563067    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 00:36:08.564777 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563783    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563790    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563793    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563796    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563798    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563801    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563804    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563807    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563810    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563812    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563815    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563818    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563820    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563823    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563826    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563828    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563831    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563834    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563837    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563840    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 00:36:08.565249 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563843    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563846    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563849    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563851    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563854    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563857    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563860    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563863    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563865    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563868    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563873    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563877    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563880    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563883    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563887    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563890    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563893    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563896    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563899    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 00:36:08.565776 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563902    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563905    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563908    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563910    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563913    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563915    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563918    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563920    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563923    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563925    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563928    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563931    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563934    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563936    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563939    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563942    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563944    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563947    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563949    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 00:36:08.566227 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563952    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563955    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563959    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563962    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563965    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563967    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563970    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563972    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563975    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563977    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563981    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563984    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563987    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563990    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563994    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563997    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.563999    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564002    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564005    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 00:36:08.566703 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564007    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564010    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564012    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564015    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564018    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564021    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564024    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564027    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564029    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564032    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564035    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564037    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564040    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564043    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564045    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564048    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564050    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564053    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564055    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564058    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 00:36:08.567156 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564060    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564063    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564459    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564465    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564468    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564471    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564474    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564477    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564480    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564483    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564486    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564489    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564492    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564495    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564497    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564500    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564502    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564506    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564509    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564511    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 00:36:08.567649 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564514    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564516    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564520    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564522    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564525    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564529    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564532    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564535    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564537    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564540    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564543    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564545    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564548    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564550    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564553    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564556    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564559    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564562    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564564    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 00:36:08.568127 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564566    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564570    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564574    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564577    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564580    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564583    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564585    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564588    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564591    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564595    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564598    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564601    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564604    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564607    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564609    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564612    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564615    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564618    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564620    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 00:36:08.568608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564623    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564625    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564628    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564631    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564634    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564636    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564639    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564641    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564644    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564646    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564649    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564652    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564654    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564657    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564659    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564662    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564665    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564667    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564670    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564672    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 00:36:08.569057 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564674    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564677    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564680    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564682    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564685    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564688    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564690    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564693    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564695    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564698    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564700    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564703    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564706    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564708    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564710    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564713    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564716    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564718    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564721    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564724    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 00:36:08.569565 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564726    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564729    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.564731    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564792    2586 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564799    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564806    2586 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564810    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564814    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564817    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564827    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564832    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564835    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564838    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564841    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564845    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564848    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564851    2586 flags.go:64] FLAG: --cgroup-root=""
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564854    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564857    2586 flags.go:64] FLAG: --client-ca-file=""
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564859    2586 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564862    2586 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564866    2586 flags.go:64] FLAG: --cluster-domain=""
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564869    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 00:36:08.570038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564872    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564876    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564879    2586 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564882    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564885    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564888    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564892    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564897    2586 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564900    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564903    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564906    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564909    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564913    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564915    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564918    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564921    2586 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564924    2586 flags.go:64] FLAG: --enable-server="true"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564926    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564931    2586 flags.go:64] FLAG: --event-burst="100"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564935    2586 flags.go:64] FLAG: --event-qps="50"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564938    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564941    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564943    2586 flags.go:64] FLAG: --eviction-hard=""
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564949    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 00:36:08.570599 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564952    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564955    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564957    2586 flags.go:64] FLAG: --eviction-soft=""
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564960    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564963    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564966    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564969    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564971    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564974    2586 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564977    2586 flags.go:64] FLAG: --feature-gates=""
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564980    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564983    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564986    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564989    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564993    2586 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.564998    2586 flags.go:64] FLAG: --help="false"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565000    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-132-227.ec2.internal"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565004    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565007    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565010    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565013    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565016    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565019    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 00:36:08.571267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565022    2586 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565025    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565028    2586 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565031    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565034    2586 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565037    2586 flags.go:64] FLAG: --kube-reserved=""
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565040    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565043    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565047    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565050    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565053    2586 flags.go:64] FLAG: --lock-file=""
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565055    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565058    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565062    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565067    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565070    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565073    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565075    2586 flags.go:64] FLAG: --logging-format="text"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565078    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565081    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565084    2586 flags.go:64] FLAG: --manifest-url=""
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565087    2586 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565091    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565094    2586 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565099    2586 flags.go:64] FLAG: --max-pods="110"
Jul 29 00:36:08.571904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565102    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565105    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565108    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565111    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565114    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565116    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565119    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565126    2586 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565130    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565133    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565136    2586 flags.go:64] FLAG: --pod-cidr=""
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565139    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565144    2586 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565147    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565151    2586 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565153    2586 flags.go:64] FLAG: --port="10250"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565158    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565161    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c6ee42c2da3c138e"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565164    2586 flags.go:64] FLAG: --qos-reserved=""
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565167    2586 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565170    2586 flags.go:64] FLAG: --register-node="true"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565173    2586 flags.go:64] FLAG: --register-with-taints=""
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565176    2586 flags.go:64] FLAG: --registry-burst="10"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565179    2586 flags.go:64] FLAG: --registry-qps="5"
Jul 29 00:36:08.572916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565182    2586 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565185    2586 flags.go:64] FLAG: --reserved-memory=""
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565188    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565191    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565194    2586 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565197    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565200    2586 flags.go:64] FLAG: --runonce="false"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565204    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565207    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565209    2586 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565212    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565215    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565218    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565221    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565224    2586 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565227    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565229    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565232    2586 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565235    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565238    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565241    2586 flags.go:64] FLAG: --system-cgroups=""
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565243    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565250    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565253    2586 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565259    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 00:36:08.573577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565264    2586 flags.go:64] FLAG: --tls-min-version=""
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565267    2586 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565270    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565272    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565275    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565278    2586 flags.go:64] FLAG: --v="2"
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565282    2586 flags.go:64] FLAG: --version="false"
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565286    2586 flags.go:64] FLAG: --vmodule=""
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565289    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.565293    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565403    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565406    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565411    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565415    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565419    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565422    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565425    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565428    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565431    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565434    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565437    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 00:36:08.574178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565440    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565455    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565458    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565461    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565464    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565467    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565470    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565473    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565475    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565478    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565482    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565485    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565487    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565491    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565495    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565497    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565500    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565503    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565505    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565508    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 00:36:08.574751 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565511    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565513    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565516    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565518    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565521    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565525    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565528    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565531    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565533    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565536    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565538    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565541    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565543    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565546    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565549    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565551    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565554    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565556    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565559    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565561    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 00:36:08.575222 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565564    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565566    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565570    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565573    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565576    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565578    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565581    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565583    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565586    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565588    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565591    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565594    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565596    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565599    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565601    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565604    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565607    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565610    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565613    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565615    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 00:36:08.575725 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565618    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565621    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565623    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565626    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565628    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565631    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565633    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565635    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565638    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565640    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565643    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565645    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565648    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565650    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565655    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565659    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565662    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565665    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565668    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 00:36:08.576196 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565670    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565673    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565676    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565678    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565681    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565684    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565686    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565689    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.565691    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.566326    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.574317    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.574332    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.574356    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.574363    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 00:36:08.576669 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574418    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574424    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574427    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574430    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574433    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574436    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574439    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574456    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574460    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574464    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574467    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574470    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574474    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574478    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574488    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574492    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574494    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574497    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574500    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574503    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 00:36:08.577004 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574505    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574508    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574510    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574513    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574517    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574520    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574522    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574526    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574528    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574531    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574534    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574537    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574540    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574542    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574545    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574547    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574550    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574552    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574555    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574557    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 00:36:08.577527 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574560    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574563    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574565    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574567    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574570    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574573    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574576    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574579    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574582    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574585    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574587    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574590    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574593    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574596    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574598    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574601    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574603    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574606    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574609    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574611    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 00:36:08.578045 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574614    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574617    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574619    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574622    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574625    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574629    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574633    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574636    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574639    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574641    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574644    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574647    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574649    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574652    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574654    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574656    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574659    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574661    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574664    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 00:36:08.578534 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574668    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574670    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574673    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574675    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574678    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574680    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574683    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574685    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574689    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574692    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574695    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574698    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574701    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574704    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574706    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574709    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574712    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574714    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574733    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 00:36:08.579086 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574736    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.574742    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574852    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574857    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574860    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574862    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574865    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574868    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574871    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574874    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574877    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574880    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574883    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574886    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574890    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 00:36:08.579872 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574893    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574896    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574899    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574902    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574904    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574907    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574909    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574912    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574914    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574917    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574920    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574922    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574925    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574928    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574931    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574933    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574936    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574939    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574941    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574944    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 00:36:08.580294 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574947    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574951    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574955    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574958    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574960    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574963    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574966    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574968    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574971    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574973    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574976    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574979    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574982    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574984    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574987    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574989    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574992    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574994    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574997    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 00:36:08.580860 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.574999    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575002    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575004    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575007    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575009    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575012    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575015    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575018    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575020    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575023    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575025    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575028    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575030    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575033    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575036    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575040    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575043    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575045    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575048    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 00:36:08.581382 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575051    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575053    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575055    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575058    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575060    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575063    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575066    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575068    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575071    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575074    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575076    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575079    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575081    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575084    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575086    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575089    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575091    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575094    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575096    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575100    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 00:36:08.581945 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575104    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575106    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575109    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575112    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575115    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575117    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575120    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:08.575123    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.575128    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.575230    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.577467    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.578800    2586 server.go:1019] "Starting client certificate rotation"
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.578917    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 00:36:08.582586 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.578967    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 00:36:08.582992 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.581647    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 00:36:08.586071 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.586053    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 00:36:08.598829 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.598811    2586 log.go:25] "Validated CRI v1 runtime API"
Jul 29 00:36:08.600895 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.600882    2586 log.go:25] "Validated CRI v1 image API"
Jul 29 00:36:08.601638 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.601625    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 00:36:08.604120 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.604096    2586 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9b37ecd2-2c2b-4da8-8cac-ccd01fa83c02:/dev/nvme0n1p4 fef975c9-e4bd-4b5f-a478-6925868102ff:/dev/nvme0n1p3]
Jul 29 00:36:08.604180 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.604119    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 00:36:08.609964 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.609843    2586 manager.go:216] Machine: {Timestamp:2026-07-29 00:36:08.608625558 +0000 UTC m=+0.335303092 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096920 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24cfae50a6914e44a9e1a5ad85d9a2 SystemUUID:ec24cfae-50a6-914e-44a9-e1a5ad85d9a2 BootID:c75ef065-775a-45ad-af05-83cd5b960166 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c0:3b:cd:9a:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c0:3b:cd:9a:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:6e:5a:33:01:37 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 00:36:08.609964 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.609954    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 00:36:08.610083 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.610034    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 00:36:08.610170 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.610154    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 00:36:08.614013 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.613987    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 00:36:08.614137 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.614015    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-227.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 00:36:08.615113 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.615103    2586 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 00:36:08.615153 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.615115    2586 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 00:36:08.615153 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.615123    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 00:36:08.615153 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.615136    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 00:36:08.615910 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.615901    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 00:36:08.618529 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.618518    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 00:36:08.618632 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.618622    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 00:36:08.618670 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.618651    2586 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 00:36:08.618670 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.618660    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 00:36:08.618741 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.618680    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 00:36:08.618741 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.618688    2586 kubelet.go:398] "Adding apiserver pod source"
Jul 29 00:36:08.618741 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.618699    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 00:36:08.619055 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.619044    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 00:36:08.619103 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.619062    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 00:36:08.621103 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.621087    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 00:36:08.622855 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.622841    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 00:36:08.622942 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.622870    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 00:36:08.625011 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625000    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 00:36:08.625011 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625015    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625021    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625026    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625035    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625040    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625048    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625060    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625066    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625072    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625080    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 00:36:08.625100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.625086    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 00:36:08.626885 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.626873    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 00:36:08.626885 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.626883    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 00:36:08.627576 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.627553    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 00:36:08.627576 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.627556    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 00:36:08.629782 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.629770    2586 server.go:1268] "Started kubelet"
Jul 29 00:36:08.629869 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.629840    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 00:36:08.630257 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.629920    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 00:36:08.630257 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.629985    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 00:36:08.630648 ip-10-0-132-227 systemd[1]: Started Kubernetes Kubelet.
Jul 29 00:36:08.631777 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.631644    2586 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 00:36:08.631777 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.631648    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 00:36:08.635777 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.635759    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mfgb8"
Jul 29 00:36:08.635976 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.635960    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 00:36:08.638133 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.638117    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-227.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 00:36:08.639092 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.638129    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-227.ec2.internal.18c69b957be0d5bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-227.ec2.internal,UID:ip-10-0-132-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-227.ec2.internal,},FirstTimestamp:2026-07-29 00:36:08.629745087 +0000 UTC m=+0.356422621,LastTimestamp:2026-07-29 00:36:08.629745087 +0000 UTC m=+0.356422621,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-227.ec2.internal,}"
Jul 29 00:36:08.639209 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.639188    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 00:36:08.639301 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.639195    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 00:36:08.639356 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.639338    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:08.640308 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640068    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 00:36:08.640308 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640071    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 00:36:08.640308 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640088    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 00:36:08.640308 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640153    2586 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 00:36:08.640308 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640163    2586 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 00:36:08.640729 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640713    2586 factory.go:153] Registering CRI-O factory
Jul 29 00:36:08.640824 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640756    2586 factory.go:223] Registration of the crio container factory successfully
Jul 29 00:36:08.640824 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640803    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 00:36:08.640824 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640812    2586 factory.go:55] Registering systemd factory
Jul 29 00:36:08.640824 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640819    2586 factory.go:223] Registration of the systemd container factory successfully
Jul 29 00:36:08.641022 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640838    2586 factory.go:103] Registering Raw factory
Jul 29 00:36:08.641022 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.640851    2586 manager.go:1196] Started watching for new ooms in manager
Jul 29 00:36:08.641289 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.641275    2586 manager.go:349] Starting recovery of all containers
Jul 29 00:36:08.642663 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.642640    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mfgb8"
Jul 29 00:36:08.643426 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.643395    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 00:36:08.643558 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.643535    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 00:36:08.653956 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.653938    2586 manager.go:354] Recovery completed
Jul 29 00:36:08.658049 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.658037    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 00:36:08.659731 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.659714    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 00:36:08.659815 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.659744    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 00:36:08.659815 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.659761    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientPID"
Jul 29 00:36:08.660930 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.660914    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 00:36:08.660930 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.660930    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 00:36:08.661025 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.660933    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 00:36:08.661025 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.660947    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 00:36:08.661681 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.661662    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 00:36:08.661742 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.661693    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 00:36:08.661742 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.661704    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientPID"
Jul 29 00:36:08.663494 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.663483    2586 policy_none.go:49] "None policy: Start"
Jul 29 00:36:08.663548 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.663498    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 00:36:08.663548 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.663509    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 00:36:08.665536 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.665516    2586 policy_none.go:47] "Start"
Jul 29 00:36:08.704756 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.704593    2586 manager.go:337] "Starting Device Plugin manager"
Jul 29 00:36:08.704821 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.704785    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 00:36:08.704821 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.704799    2586 server.go:87] "Starting device plugin registration server"
Jul 29 00:36:08.705052 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.705041    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 00:36:08.705081 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.705055    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 00:36:08.705175 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.705153    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 00:36:08.705270 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.705236    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 00:36:08.705270 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.705245    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 00:36:08.705929 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.705906    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 00:36:08.705995 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.705970    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:08.796106 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.796067    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 00:36:08.798307 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.797244    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 00:36:08.798307 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.797278    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 00:36:08.798307 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.797307    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 00:36:08.798307 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.797340    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 00:36:08.800336 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.800314    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 00:36:08.805576 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.805537    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 00:36:08.806350 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.806328    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 00:36:08.806431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.806362    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 00:36:08.806431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.806375    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientPID"
Jul 29 00:36:08.806431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.806400    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-227.ec2.internal"
Jul 29 00:36:08.814794 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.814777    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-227.ec2.internal"
Jul 29 00:36:08.814862 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.814800    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-227.ec2.internal\": node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:08.830711 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.830689    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:08.897878 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.897821    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal"]
Jul 29 00:36:08.898012 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.897916    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 00:36:08.899579 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.899563    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 00:36:08.899676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.899590    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 00:36:08.899676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.899601    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientPID"
Jul 29 00:36:08.901864 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.901852    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 00:36:08.902030 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.902009    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:08.902072 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.902051    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 00:36:08.902594 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.902580    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 00:36:08.902666 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.902610    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 00:36:08.902666 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.902621    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientPID"
Jul 29 00:36:08.903246 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.903232    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 00:36:08.903307 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.903260    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 00:36:08.903307 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.903272    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientPID"
Jul 29 00:36:08.905435 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.905420    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:08.905517 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.905453    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 00:36:08.906099 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.906081    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 00:36:08.906196 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.906110    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 00:36:08.906196 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:08.906120    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeHasSufficientPID"
Jul 29 00:36:08.927348 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.927319    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-227.ec2.internal\" not found" node="ip-10-0-132-227.ec2.internal"
Jul 29 00:36:08.930757 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.930740    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:08.931996 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:08.931980    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-227.ec2.internal\" not found" node="ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.449365 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:10.449328    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:10.459310 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.459278    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5807e7c8a4e7f09ca39bb57b7b6d746e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal\" (UID: \"5807e7c8a4e7f09ca39bb57b7b6d746e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.459310 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.459313    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5807e7c8a4e7f09ca39bb57b7b6d746e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal\" (UID: \"5807e7c8a4e7f09ca39bb57b7b6d746e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.459479 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.459332    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f053b035387553a0f709b762fddad9a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-227.ec2.internal\" (UID: \"8f053b035387553a0f709b762fddad9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.549767 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:10.549681    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:10.560038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.560009    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5807e7c8a4e7f09ca39bb57b7b6d746e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal\" (UID: \"5807e7c8a4e7f09ca39bb57b7b6d746e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.560115 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.560043    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5807e7c8a4e7f09ca39bb57b7b6d746e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal\" (UID: \"5807e7c8a4e7f09ca39bb57b7b6d746e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.560115 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.560065    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f053b035387553a0f709b762fddad9a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-227.ec2.internal\" (UID: \"8f053b035387553a0f709b762fddad9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.560115 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.560101    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5807e7c8a4e7f09ca39bb57b7b6d746e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal\" (UID: \"5807e7c8a4e7f09ca39bb57b7b6d746e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.560283 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.560117    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5807e7c8a4e7f09ca39bb57b7b6d746e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal\" (UID: \"5807e7c8a4e7f09ca39bb57b7b6d746e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.560283 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.560101    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f053b035387553a0f709b762fddad9a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-227.ec2.internal\" (UID: \"8f053b035387553a0f709b762fddad9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.648271 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.648220    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.650338 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:10.650321    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:10.653429 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.653221    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:10.750772 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:10.750736    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:10.851257 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:10.851236    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:10.863968 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.863936    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 00:36:10.951646 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:10.951608    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:10.996238 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.996192    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 00:36:10.996702 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.996356    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 00:36:10.996702 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:10.996389    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 00:36:11.052703 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:11.052664    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:11.067054 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:11.067015    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 00:31:10 +0000 UTC" deadline="2028-03-12 06:03:37.190583689 +0000 UTC"
Jul 29 00:36:11.067054 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:11.067045    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14213h27m26.123541649s"
Jul 29 00:36:11.153460 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:11.153393    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:11.254369 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:11.254338    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:11.327414 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:11.327370    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f053b035387553a0f709b762fddad9a.slice/crio-d4004ea628eff754ab560d10eae94d4fa31176bdb0f61f778c8a31f89b50ba8b WatchSource:0}: Error finding container d4004ea628eff754ab560d10eae94d4fa31176bdb0f61f778c8a31f89b50ba8b: Status 404 returned error can't find the container with id d4004ea628eff754ab560d10eae94d4fa31176bdb0f61f778c8a31f89b50ba8b
Jul 29 00:36:11.328022 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:11.327994    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5807e7c8a4e7f09ca39bb57b7b6d746e.slice/crio-e209e6295ceaa5c048ef4fcba64ab73adc3178f18f64ce52be11f8646df1d4d9 WatchSource:0}: Error finding container e209e6295ceaa5c048ef4fcba64ab73adc3178f18f64ce52be11f8646df1d4d9: Status 404 returned error can't find the container with id e209e6295ceaa5c048ef4fcba64ab73adc3178f18f64ce52be11f8646df1d4d9
Jul 29 00:36:11.335076 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:11.335060    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 00:36:11.354475 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:11.354441    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-227.ec2.internal\" not found"
Jul 29 00:36:11.396819 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:11.396793    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 00:36:11.457224 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:11.457143    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:11.468091 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:11.468074    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 00:36:11.468900 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:11.468887    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal"
Jul 29 00:36:11.479336 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:11.479315    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 00:36:11.582685 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:11.582649    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 00:36:12.038258 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.038221    2586 apiserver.go:52] "Watching apiserver"
Jul 29 00:36:12.046676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.046645    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 00:36:12.049244 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.049207    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-2ljbv","openshift-ovn-kubernetes/ovnkube-node-llvgq","kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr","openshift-dns/node-resolver-krk2j","openshift-image-registry/node-ca-74znh","openshift-multus/multus-57bk9","openshift-network-diagnostics/network-check-target-m2vg8","openshift-network-operator/iptables-alerter-zg9n6","kube-system/global-pull-secret-syncer-hrq9b","kube-system/konnectivity-agent-2t4gv","openshift-cluster-node-tuning-operator/tuned-jchqv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal","openshift-multus/multus-additional-cni-plugins-bbnr9"]
Jul 29 00:36:12.052000 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.051899    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.054305 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.054092    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:12.055718 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.055150    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q2jkw\""
Jul 29 00:36:12.055718 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.055241    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 00:36:12.055718 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.055426    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 00:36:12.055718 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.055431    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 00:36:12.055718 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.055633    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 00:36:12.056782 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.056748    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.056902 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.056855    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 00:36:12.056974 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.056953    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xf4fn\""
Jul 29 00:36:12.057030 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.056863    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 00:36:12.057096 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.057081    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 00:36:12.059064 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.058932    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.059169 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.059152    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 00:36:12.059461 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.059330    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-24qj5\""
Jul 29 00:36:12.059461 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.059403    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 00:36:12.061433 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.061288    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 00:36:12.061677 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.061660    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.062067 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.062045    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 00:36:12.062159 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.062075    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z9npk\""
Jul 29 00:36:12.062159 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.062130    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 00:36:12.063497 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.063476    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.064042 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.064022    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 00:36:12.064519 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.064499    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 00:36:12.064759 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.064615    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-swbgk\""
Jul 29 00:36:12.065786 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.065767    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.066931 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.066911    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 00:36:12.067145 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.067130    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 00:36:12.068206 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.068136    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rc7q8\""
Jul 29 00:36:12.068206 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.068152    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069124    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-socket-dir-parent\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069170    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/acbd27bf-81aa-49e9-9c59-d58bcf765cd8-konnectivity-ca\") pod \"konnectivity-agent-2t4gv\" (UID: \"acbd27bf-81aa-49e9-9c59-d58bcf765cd8\") " pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069230    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4442676d-6797-4d73-bdde-af2342f528e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069259    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4442676d-6797-4d73-bdde-af2342f528e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069286    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4442676d-6797-4d73-bdde-af2342f528e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069317    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjhg\" (UniqueName: \"kubernetes.io/projected/c308d7f1-f1da-4328-895a-e3790e8ca866-kube-api-access-4kjhg\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069345    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-run-k8s-cni-cncf-io\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069374    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-hostroot\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069405    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-daemon-config\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069430    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-run-multus-certs\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069459    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069488    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfqh\" (UniqueName: \"kubernetes.io/projected/4442676d-6797-4d73-bdde-af2342f528e5-kube-api-access-xzfqh\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069517    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069546    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-device-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069572    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5536843-3e0d-4cbb-bfa8-938694ab3e63-cni-binary-copy\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.069913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069576    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 00:36:12.070723 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069601    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-var-lib-cni-multus\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.070723 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069651    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8lhn\" (UniqueName: \"kubernetes.io/projected/f5536843-3e0d-4cbb-bfa8-938694ab3e63-kube-api-access-d8lhn\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.070723 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069691    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-etc-selinux\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.070723 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069718    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-cni-dir\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.070723 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069746    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-var-lib-cni-bin\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.070723 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069797    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-conf-dir\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.070723 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.069834    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/acbd27bf-81aa-49e9-9c59-d58bcf765cd8-agent-certs\") pod \"konnectivity-agent-2t4gv\" (UID: \"acbd27bf-81aa-49e9-9c59-d58bcf765cd8\") " pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:12.070723 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.070603    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:12.071112 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.070782    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:12.071112 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.070887    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 00:36:12.071112 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.070972    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7xh2q\""
Jul 29 00:36:12.071299 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071278    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 00:36:12.071627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071340    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 00:36:12.071719 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071623    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-system-cni-dir\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.071719 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071667    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-run-netns\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.071837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071718    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-system-cni-dir\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.071837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071765    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-registration-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.071837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071805    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-cnibin\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.072003 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071856    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-cnibin\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.072003 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071896    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-os-release\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.072003 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.071949    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-socket-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.072121 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.072032    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-os-release\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.072121 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.072085    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-var-lib-kubelet\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.072226 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.072125    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-etc-kubernetes\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.072226 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.072151    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-sys-fs\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.074000 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.073244    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.076264 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.075697    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.076264 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.075764    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:12.076264 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.076021    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mwpcf\""
Jul 29 00:36:12.076264 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.076023    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 00:36:12.077567 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.077157    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 00:36:12.077567 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.077163    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 00:36:12.077567 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.077280    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 00:36:12.077567 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.077480    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 00:36:12.077567 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.077524    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 00:36:12.078904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.078804    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:12.078904 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.078857    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:12.081710 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.081279    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.084159 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.083925    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 00:36:12.084159 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.083933    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 00:36:12.084159 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.083972    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tv86b\""
Jul 29 00:36:12.103755 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.103725    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lvxkq"
Jul 29 00:36:12.113676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.113643    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lvxkq"
Jul 29 00:36:12.136753 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.136567    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 00:36:12.159256 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.159218    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 00:36:12.173237 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173189    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-cnibin\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.173402 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173244    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-run-openvswitch\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.173402 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173272    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.173402 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173296    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vcb\" (UniqueName: \"kubernetes.io/projected/b06e3b5d-f144-4590-ad28-80c6a76acc5e-kube-api-access-89vcb\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.173402 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173304    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-cnibin\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.173402 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173323    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-var-lib-kubelet\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.173402 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173359    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7a2ae25-317e-403b-a257-b993fe994cf2-serviceca\") pod \"node-ca-74znh\" (UID: \"e7a2ae25-317e-403b-a257-b993fe994cf2\") " pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.173402 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173396    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-socket-dir-parent\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4442676d-6797-4d73-bdde-af2342f528e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173468    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7a2ae25-317e-403b-a257-b993fe994cf2-host\") pod \"node-ca-74znh\" (UID: \"e7a2ae25-317e-403b-a257-b993fe994cf2\") " pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173516    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6vt\" (UniqueName: \"kubernetes.io/projected/f1bad3c0-d369-4188-b5d2-12090d1718e6-kube-api-access-vt6vt\") pod \"iptables-alerter-zg9n6\" (UID: \"f1bad3c0-d369-4188-b5d2-12090d1718e6\") " pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173548    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-socket-dir-parent\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173562    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-daemon-config\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173601    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-run-multus-certs\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173660    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173685    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-device-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173712    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-etc-selinux\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173741    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-run-netns\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.173793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173779    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-var-lib-openvswitch\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173803    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-run-ovn\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173853    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-var-lib-kubelet\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5536843-3e0d-4cbb-bfa8-938694ab3e63-cni-binary-copy\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173888    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-cni-netd\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173907    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-run-multus-certs\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173913    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7gmm\" (UniqueName: \"kubernetes.io/projected/e7a2ae25-317e-403b-a257-b993fe994cf2-kube-api-access-v7gmm\") pod \"node-ca-74znh\" (UID: \"e7a2ae25-317e-403b-a257-b993fe994cf2\") " pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173937    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5c14500d-ea5d-4433-ac3c-2df82c0ef982-dbus\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173961    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-sysctl-conf\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.173982    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-run\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174010    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-lib-modules\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174077    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-var-lib-cni-bin\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174102    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-conf-dir\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174126    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/acbd27bf-81aa-49e9-9c59-d58bcf765cd8-agent-certs\") pod \"konnectivity-agent-2t4gv\" (UID: \"acbd27bf-81aa-49e9-9c59-d58bcf765cd8\") " pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174165    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-systemd-units\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-slash\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174234    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-system-cni-dir\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.174361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174242    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174261    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174288    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b06e3b5d-f144-4590-ad28-80c6a76acc5e-ovn-node-metrics-cert\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174291    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-device-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174312    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b06e3b5d-f144-4590-ad28-80c6a76acc5e-ovnkube-script-lib\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174168    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4442676d-6797-4d73-bdde-af2342f528e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174387    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-etc-selinux\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174416    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-kubernetes\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174442    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a115e159-a003-4bdc-8e43-94ac470b5d1a-hosts-file\") pod \"node-resolver-krk2j\" (UID: \"a115e159-a003-4bdc-8e43-94ac470b5d1a\") " pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174468    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-run-netns\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174476    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-daemon-config\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174494    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-system-cni-dir\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174520    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-registration-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174547    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-log-socket\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174574    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2w9v\" (UniqueName: \"kubernetes.io/projected/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-kube-api-access-f2w9v\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174599    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1bad3c0-d369-4188-b5d2-12090d1718e6-host-slash\") pod \"iptables-alerter-zg9n6\" (UID: \"f1bad3c0-d369-4188-b5d2-12090d1718e6\") " pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.175141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174623    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174694    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f766f703-d84b-4ae3-9e3a-c53ede205ab8-tmp\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174718    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-cnibin\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174742    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-os-release\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174748    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5536843-3e0d-4cbb-bfa8-938694ab3e63-cni-binary-copy\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174766    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-socket-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174798    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-node-log\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174785    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174811    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-run-netns\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174824    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5c14500d-ea5d-4433-ac3c-2df82c0ef982-kubelet-config\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174836    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-cnibin\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174850    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-os-release\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174861    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-system-cni-dir\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-etc-kubernetes\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174889    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-registration-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174896    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-conf-dir\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174934    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-system-cni-dir\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174935    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-sys-fs\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.175986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174964    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-os-release\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174973    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-var-lib-cni-bin\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.174976    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-sys-fs\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175005    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-etc-openvswitch\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175011    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-etc-kubernetes\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175037    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b06e3b5d-f144-4590-ad28-80c6a76acc5e-ovnkube-config\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175065    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-os-release\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175122    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/acbd27bf-81aa-49e9-9c59-d58bcf765cd8-konnectivity-ca\") pod \"konnectivity-agent-2t4gv\" (UID: \"acbd27bf-81aa-49e9-9c59-d58bcf765cd8\") " pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175133    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c308d7f1-f1da-4328-895a-e3790e8ca866-socket-dir\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175167    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4442676d-6797-4d73-bdde-af2342f528e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175237    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4442676d-6797-4d73-bdde-af2342f528e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175277    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjhg\" (UniqueName: \"kubernetes.io/projected/c308d7f1-f1da-4328-895a-e3790e8ca866-kube-api-access-4kjhg\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175315    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-sysctl-d\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175340    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-tuned\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175377    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-run-k8s-cni-cncf-io\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175405    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-hostroot\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175430    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.176859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfqh\" (UniqueName: \"kubernetes.io/projected/4442676d-6797-4d73-bdde-af2342f528e5-kube-api-access-xzfqh\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175481    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-var-lib-kubelet\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175526    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a115e159-a003-4bdc-8e43-94ac470b5d1a-tmp-dir\") pod \"node-resolver-krk2j\" (UID: \"a115e159-a003-4bdc-8e43-94ac470b5d1a\") " pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175554    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-var-lib-cni-multus\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175580    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8lhn\" (UniqueName: \"kubernetes.io/projected/f5536843-3e0d-4cbb-bfa8-938694ab3e63-kube-api-access-d8lhn\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175605    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-run-systemd\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175630    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175653    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-sysconfig\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175654    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/acbd27bf-81aa-49e9-9c59-d58bcf765cd8-konnectivity-ca\") pod \"konnectivity-agent-2t4gv\" (UID: \"acbd27bf-81aa-49e9-9c59-d58bcf765cd8\") " pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175675    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-sys\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175699    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-host\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175697    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4442676d-6797-4d73-bdde-af2342f528e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175837    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-cni-dir\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175867    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175893    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1bad3c0-d369-4188-b5d2-12090d1718e6-iptables-alerter-script\") pod \"iptables-alerter-zg9n6\" (UID: \"f1bad3c0-d369-4188-b5d2-12090d1718e6\") " pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175916    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-systemd\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175943    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldvk\" (UniqueName: \"kubernetes.io/projected/f766f703-d84b-4ae3-9e3a-c53ede205ab8-kube-api-access-6ldvk\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.177592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175971    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvvf\" (UniqueName: \"kubernetes.io/projected/a115e159-a003-4bdc-8e43-94ac470b5d1a-kube-api-access-mxvvf\") pod \"node-resolver-krk2j\" (UID: \"a115e159-a003-4bdc-8e43-94ac470b5d1a\") " pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175986    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-var-lib-cni-multus\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.175995    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-kubelet\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.176021    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-cni-bin\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.176093    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4442676d-6797-4d73-bdde-af2342f528e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.176175    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b06e3b5d-f144-4590-ad28-80c6a76acc5e-env-overrides\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.176219    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-modprobe-d\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.176349    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-hostroot\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.176397    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-host-run-k8s-cni-cncf-io\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.176532    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4442676d-6797-4d73-bdde-af2342f528e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.178356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.176654    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5536843-3e0d-4cbb-bfa8-938694ab3e63-multus-cni-dir\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.178845 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.178815    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/acbd27bf-81aa-49e9-9c59-d58bcf765cd8-agent-certs\") pod \"konnectivity-agent-2t4gv\" (UID: \"acbd27bf-81aa-49e9-9c59-d58bcf765cd8\") " pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:12.187521 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.187463    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8lhn\" (UniqueName: \"kubernetes.io/projected/f5536843-3e0d-4cbb-bfa8-938694ab3e63-kube-api-access-d8lhn\") pod \"multus-57bk9\" (UID: \"f5536843-3e0d-4cbb-bfa8-938694ab3e63\") " pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.187917 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.187884    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfqh\" (UniqueName: \"kubernetes.io/projected/4442676d-6797-4d73-bdde-af2342f528e5-kube-api-access-xzfqh\") pod \"multus-additional-cni-plugins-bbnr9\" (UID: \"4442676d-6797-4d73-bdde-af2342f528e5\") " pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.189541 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.189445    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjhg\" (UniqueName: \"kubernetes.io/projected/c308d7f1-f1da-4328-895a-e3790e8ca866-kube-api-access-4kjhg\") pod \"aws-ebs-csi-driver-node-8s6cr\" (UID: \"c308d7f1-f1da-4328-895a-e3790e8ca866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.224605 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.224537    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal" event={"ID":"8f053b035387553a0f709b762fddad9a","Type":"ContainerStarted","Data":"d4004ea628eff754ab560d10eae94d4fa31176bdb0f61f778c8a31f89b50ba8b"}
Jul 29 00:36:12.226514 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.226478    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal" event={"ID":"5807e7c8a4e7f09ca39bb57b7b6d746e","Type":"ContainerStarted","Data":"e209e6295ceaa5c048ef4fcba64ab73adc3178f18f64ce52be11f8646df1d4d9"}
Jul 29 00:36:12.277257 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277222    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b06e3b5d-f144-4590-ad28-80c6a76acc5e-ovnkube-script-lib\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.277257 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277260    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-kubernetes\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.277499 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277284    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a115e159-a003-4bdc-8e43-94ac470b5d1a-hosts-file\") pod \"node-resolver-krk2j\" (UID: \"a115e159-a003-4bdc-8e43-94ac470b5d1a\") " pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.277499 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277359    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a115e159-a003-4bdc-8e43-94ac470b5d1a-hosts-file\") pod \"node-resolver-krk2j\" (UID: \"a115e159-a003-4bdc-8e43-94ac470b5d1a\") " pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.277499 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277361    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-kubernetes\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.277499 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277407    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-log-socket\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.277499 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277433    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2w9v\" (UniqueName: \"kubernetes.io/projected/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-kube-api-access-f2w9v\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:12.277499 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277462    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1bad3c0-d369-4188-b5d2-12090d1718e6-host-slash\") pod \"iptables-alerter-zg9n6\" (UID: \"f1bad3c0-d369-4188-b5d2-12090d1718e6\") " pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.277499 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277476    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-log-socket\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.277499 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277487    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277511    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f766f703-d84b-4ae3-9e3a-c53ede205ab8-tmp\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277537    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-node-log\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277561    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5c14500d-ea5d-4433-ac3c-2df82c0ef982-kubelet-config\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277588    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-etc-openvswitch\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277611    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b06e3b5d-f144-4590-ad28-80c6a76acc5e-ovnkube-config\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277639    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-sysctl-d\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277681    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-tuned\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277709    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-var-lib-kubelet\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277724    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-node-log\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277769    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-var-lib-kubelet\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277798    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5c14500d-ea5d-4433-ac3c-2df82c0ef982-kubelet-config\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277805    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-etc-openvswitch\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.277860 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.277841    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277866    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a115e159-a003-4bdc-8e43-94ac470b5d1a-tmp-dir\") pod \"node-resolver-krk2j\" (UID: \"a115e159-a003-4bdc-8e43-94ac470b5d1a\") " pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277885    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b06e3b5d-f144-4590-ad28-80c6a76acc5e-ovnkube-script-lib\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277898    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-sysctl-d\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277900    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-run-systemd\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277843    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1bad3c0-d369-4188-b5d2-12090d1718e6-host-slash\") pod \"iptables-alerter-zg9n6\" (UID: \"f1bad3c0-d369-4188-b5d2-12090d1718e6\") " pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.277923    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret podName:5c14500d-ea5d-4433-ac3c-2df82c0ef982 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:12.777902647 +0000 UTC m=+3.086830488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret") pod "global-pull-secret-syncer-hrq9b" (UID: "5c14500d-ea5d-4433-ac3c-2df82c0ef982") : object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277928    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-run-systemd\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.277998    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-sysconfig\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-sysconfig\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.278071    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278128    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a115e159-a003-4bdc-8e43-94ac470b5d1a-tmp-dir\") pod \"node-resolver-krk2j\" (UID: \"a115e159-a003-4bdc-8e43-94ac470b5d1a\") " pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.278149    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs podName:3c12d6fa-daae-4fc8-9325-1ae4a62765e9 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:12.7781301 +0000 UTC m=+3.087057937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs") pod "network-metrics-daemon-2ljbv" (UID: "3c12d6fa-daae-4fc8-9325-1ae4a62765e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278170    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-sys\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278215    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-host\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278237    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-sys\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.278455 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278243    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278275    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1bad3c0-d369-4188-b5d2-12090d1718e6-iptables-alerter-script\") pod \"iptables-alerter-zg9n6\" (UID: \"f1bad3c0-d369-4188-b5d2-12090d1718e6\") " pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278301    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-host\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-systemd\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278340    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldvk\" (UniqueName: \"kubernetes.io/projected/f766f703-d84b-4ae3-9e3a-c53ede205ab8-kube-api-access-6ldvk\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278368    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvvf\" (UniqueName: \"kubernetes.io/projected/a115e159-a003-4bdc-8e43-94ac470b5d1a-kube-api-access-mxvvf\") pod \"node-resolver-krk2j\" (UID: \"a115e159-a003-4bdc-8e43-94ac470b5d1a\") " pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-kubelet\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278413    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-cni-bin\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278435    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b06e3b5d-f144-4590-ad28-80c6a76acc5e-env-overrides\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278458    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-modprobe-d\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278484    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-run-openvswitch\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278507    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278498    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-kubelet\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278694    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vcb\" (UniqueName: \"kubernetes.io/projected/b06e3b5d-f144-4590-ad28-80c6a76acc5e-kube-api-access-89vcb\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278730    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7a2ae25-317e-403b-a257-b993fe994cf2-serviceca\") pod \"node-ca-74znh\" (UID: \"e7a2ae25-317e-403b-a257-b993fe994cf2\") " pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278758    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7a2ae25-317e-403b-a257-b993fe994cf2-host\") pod \"node-ca-74znh\" (UID: \"e7a2ae25-317e-403b-a257-b993fe994cf2\") " pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278784    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6vt\" (UniqueName: \"kubernetes.io/projected/f1bad3c0-d369-4188-b5d2-12090d1718e6-kube-api-access-vt6vt\") pod \"iptables-alerter-zg9n6\" (UID: \"f1bad3c0-d369-4188-b5d2-12090d1718e6\") " pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.279334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278838    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-run-openvswitch\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278863    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b06e3b5d-f144-4590-ad28-80c6a76acc5e-ovnkube-config\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278872    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-run-netns\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278880    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b06e3b5d-f144-4590-ad28-80c6a76acc5e-env-overrides\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278863    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1bad3c0-d369-4188-b5d2-12090d1718e6-iptables-alerter-script\") pod \"iptables-alerter-zg9n6\" (UID: \"f1bad3c0-d369-4188-b5d2-12090d1718e6\") " pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278900    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-var-lib-openvswitch\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278904    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-systemd\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278935    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278956    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-var-lib-openvswitch\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278938    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-run-ovn\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278974    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-cni-bin\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.278991    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-cni-netd\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279007    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-run-netns\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279019    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7gmm\" (UniqueName: \"kubernetes.io/projected/e7a2ae25-317e-403b-a257-b993fe994cf2-kube-api-access-v7gmm\") pod \"node-ca-74znh\" (UID: \"e7a2ae25-317e-403b-a257-b993fe994cf2\") " pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279022    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7a2ae25-317e-403b-a257-b993fe994cf2-host\") pod \"node-ca-74znh\" (UID: \"e7a2ae25-317e-403b-a257-b993fe994cf2\") " pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279044    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5c14500d-ea5d-4433-ac3c-2df82c0ef982-dbus\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279050    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-cni-netd\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-sysctl-conf\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.280119 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279099    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-run\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279108    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5c14500d-ea5d-4433-ac3c-2df82c0ef982-dbus\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279122    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-lib-modules\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279149    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-systemd-units\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279154    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-run-ovn\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279173    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-slash\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279215    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b06e3b5d-f144-4590-ad28-80c6a76acc5e-ovn-node-metrics-cert\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279262    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-run\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279281    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-lib-modules\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279262    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-sysctl-conf\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279322    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-systemd-units\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279326    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279346    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7a2ae25-317e-403b-a257-b993fe994cf2-serviceca\") pod \"node-ca-74znh\" (UID: \"e7a2ae25-317e-403b-a257-b993fe994cf2\") " pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279360    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b06e3b5d-f144-4590-ad28-80c6a76acc5e-host-slash\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.279449    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-modprobe-d\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.281150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.280672    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f766f703-d84b-4ae3-9e3a-c53ede205ab8-tmp\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.281895 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.281227    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f766f703-d84b-4ae3-9e3a-c53ede205ab8-etc-tuned\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.285429 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.284551    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b06e3b5d-f144-4590-ad28-80c6a76acc5e-ovn-node-metrics-cert\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.287681 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.287651    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 00:36:12.287805 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.287686    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 00:36:12.287805 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.287700    2586 projected.go:196] Error preparing data for projected volume kube-api-access-blj2s for pod openshift-network-diagnostics/network-check-target-m2vg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:12.287805 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.287776    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s podName:4b367dde-86b2-4108-9650-90d1d9b8e47a nodeName:}" failed. No retries permitted until 2026-07-29 00:36:12.787759236 +0000 UTC m=+3.096687070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-blj2s" (UniqueName: "kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s") pod "network-check-target-m2vg8" (UID: "4b367dde-86b2-4108-9650-90d1d9b8e47a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:12.290911 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.290848    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldvk\" (UniqueName: \"kubernetes.io/projected/f766f703-d84b-4ae3-9e3a-c53ede205ab8-kube-api-access-6ldvk\") pod \"tuned-jchqv\" (UID: \"f766f703-d84b-4ae3-9e3a-c53ede205ab8\") " pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.291031 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.291018    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vcb\" (UniqueName: \"kubernetes.io/projected/b06e3b5d-f144-4590-ad28-80c6a76acc5e-kube-api-access-89vcb\") pod \"ovnkube-node-llvgq\" (UID: \"b06e3b5d-f144-4590-ad28-80c6a76acc5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.291432 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.291409    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7gmm\" (UniqueName: \"kubernetes.io/projected/e7a2ae25-317e-403b-a257-b993fe994cf2-kube-api-access-v7gmm\") pod \"node-ca-74znh\" (UID: \"e7a2ae25-317e-403b-a257-b993fe994cf2\") " pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.291736 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.291713    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2w9v\" (UniqueName: \"kubernetes.io/projected/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-kube-api-access-f2w9v\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:12.292732 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.292694    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvvf\" (UniqueName: \"kubernetes.io/projected/a115e159-a003-4bdc-8e43-94ac470b5d1a-kube-api-access-mxvvf\") pod \"node-resolver-krk2j\" (UID: \"a115e159-a003-4bdc-8e43-94ac470b5d1a\") " pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.292905 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.292883    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6vt\" (UniqueName: \"kubernetes.io/projected/f1bad3c0-d369-4188-b5d2-12090d1718e6-kube-api-access-vt6vt\") pod \"iptables-alerter-zg9n6\" (UID: \"f1bad3c0-d369-4188-b5d2-12090d1718e6\") " pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.366145 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.366110    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-57bk9"
Jul 29 00:36:12.377031 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.376842    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:12.377393 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:12.377363    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5536843_3e0d_4cbb_bfa8_938694ab3e63.slice/crio-9dba0f6f02a56c8e238b9c5dafa910fd60fc833399fd4ea5341cfffef2e0af9b WatchSource:0}: Error finding container 9dba0f6f02a56c8e238b9c5dafa910fd60fc833399fd4ea5341cfffef2e0af9b: Status 404 returned error can't find the container with id 9dba0f6f02a56c8e238b9c5dafa910fd60fc833399fd4ea5341cfffef2e0af9b
Jul 29 00:36:12.386459 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.386434    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bbnr9"
Jul 29 00:36:12.387128 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:12.387102    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbd27bf_81aa_49e9_9c59_d58bcf765cd8.slice/crio-d56a8acd01f3a156569ecb8390e156f7df035fa6cac0f247515087fa03874653 WatchSource:0}: Error finding container d56a8acd01f3a156569ecb8390e156f7df035fa6cac0f247515087fa03874653: Status 404 returned error can't find the container with id d56a8acd01f3a156569ecb8390e156f7df035fa6cac0f247515087fa03874653
Jul 29 00:36:12.394098 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.394075    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr"
Jul 29 00:36:12.397098 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:12.397074    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4442676d_6797_4d73_bdde_af2342f528e5.slice/crio-f45732eee987832a683c67d38d2437f2b9860bdded311419015b9a33d845a9b5 WatchSource:0}: Error finding container f45732eee987832a683c67d38d2437f2b9860bdded311419015b9a33d845a9b5: Status 404 returned error can't find the container with id f45732eee987832a683c67d38d2437f2b9860bdded311419015b9a33d845a9b5
Jul 29 00:36:12.401948 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.401925    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-krk2j"
Jul 29 00:36:12.403158 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:12.403132    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc308d7f1_f1da_4328_895a_e3790e8ca866.slice/crio-1b2cc3fc290be3143e780cd5a968eca7f1de1b825f5df160d8e071bf5711265f WatchSource:0}: Error finding container 1b2cc3fc290be3143e780cd5a968eca7f1de1b825f5df160d8e071bf5711265f: Status 404 returned error can't find the container with id 1b2cc3fc290be3143e780cd5a968eca7f1de1b825f5df160d8e071bf5711265f
Jul 29 00:36:12.410316 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.410297    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-74znh"
Jul 29 00:36:12.419263 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.419230    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zg9n6"
Jul 29 00:36:12.427264 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.427236    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:12.432283 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.432262    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jchqv"
Jul 29 00:36:12.782595 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.782553    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:12.782778 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.782635    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:12.782778 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.782701    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:12.782778 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.782737    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:12.782778 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.782764    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs podName:3c12d6fa-daae-4fc8-9325-1ae4a62765e9 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:13.782747868 +0000 UTC m=+4.091675688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs") pod "network-metrics-daemon-2ljbv" (UID: "3c12d6fa-daae-4fc8-9325-1ae4a62765e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:12.782920 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.782790    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret podName:5c14500d-ea5d-4433-ac3c-2df82c0ef982 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:13.782776296 +0000 UTC m=+4.091704119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret") pod "global-pull-secret-syncer-hrq9b" (UID: "5c14500d-ea5d-4433-ac3c-2df82c0ef982") : object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:12.883322 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:12.883178    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:12.883486 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.883347    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 00:36:12.883486 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.883367    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 00:36:12.883486 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.883381    2586 projected.go:196] Error preparing data for projected volume kube-api-access-blj2s for pod openshift-network-diagnostics/network-check-target-m2vg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:12.883486 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:12.883441    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s podName:4b367dde-86b2-4108-9650-90d1d9b8e47a nodeName:}" failed. No retries permitted until 2026-07-29 00:36:13.883423747 +0000 UTC m=+4.192351588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-blj2s" (UniqueName: "kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s") pod "network-check-target-m2vg8" (UID: "4b367dde-86b2-4108-9650-90d1d9b8e47a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:13.117965 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.117922    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 00:31:12 +0000 UTC" deadline="2028-02-02 09:32:50.32919009 +0000 UTC"
Jul 29 00:36:13.117965 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.117963    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13280h56m37.211232062s"
Jul 29 00:36:13.229701 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.229614    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr" event={"ID":"c308d7f1-f1da-4328-895a-e3790e8ca866","Type":"ContainerStarted","Data":"1b2cc3fc290be3143e780cd5a968eca7f1de1b825f5df160d8e071bf5711265f"}
Jul 29 00:36:13.230955 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.230916    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbnr9" event={"ID":"4442676d-6797-4d73-bdde-af2342f528e5","Type":"ContainerStarted","Data":"f45732eee987832a683c67d38d2437f2b9860bdded311419015b9a33d845a9b5"}
Jul 29 00:36:13.232564 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.232491    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2t4gv" event={"ID":"acbd27bf-81aa-49e9-9c59-d58bcf765cd8","Type":"ContainerStarted","Data":"d56a8acd01f3a156569ecb8390e156f7df035fa6cac0f247515087fa03874653"}
Jul 29 00:36:13.234883 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.234840    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-57bk9" event={"ID":"f5536843-3e0d-4cbb-bfa8-938694ab3e63","Type":"ContainerStarted","Data":"9dba0f6f02a56c8e238b9c5dafa910fd60fc833399fd4ea5341cfffef2e0af9b"}
Jul 29 00:36:13.595897 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.595867    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 00:36:13.729212 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:13.727821    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bad3c0_d369_4188_b5d2_12090d1718e6.slice/crio-6fef2e653bf51bacf9d591759539a95047c93043210bf5abcf9fbf4c4c7c31de WatchSource:0}: Error finding container 6fef2e653bf51bacf9d591759539a95047c93043210bf5abcf9fbf4c4c7c31de: Status 404 returned error can't find the container with id 6fef2e653bf51bacf9d591759539a95047c93043210bf5abcf9fbf4c4c7c31de
Jul 29 00:36:13.730678 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:13.730598    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06e3b5d_f144_4590_ad28_80c6a76acc5e.slice/crio-a95811200327abcf14135dc34150023c0a937343b1db2ffb9c229caae92be0df WatchSource:0}: Error finding container a95811200327abcf14135dc34150023c0a937343b1db2ffb9c229caae92be0df: Status 404 returned error can't find the container with id a95811200327abcf14135dc34150023c0a937343b1db2ffb9c229caae92be0df
Jul 29 00:36:13.733748 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:36:13.733263    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a2ae25_317e_403b_a257_b993fe994cf2.slice/crio-01b130a9ccd4a6acacee21f4f9c791f66bf1b68570ade42e29812723484ce510 WatchSource:0}: Error finding container 01b130a9ccd4a6acacee21f4f9c791f66bf1b68570ade42e29812723484ce510: Status 404 returned error can't find the container with id 01b130a9ccd4a6acacee21f4f9c791f66bf1b68570ade42e29812723484ce510
Jul 29 00:36:13.789387 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.789351    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:13.789533 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.789416    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:13.789597 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:13.789559    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:13.789672 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:13.789622    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs podName:3c12d6fa-daae-4fc8-9325-1ae4a62765e9 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:15.789601942 +0000 UTC m=+6.098529767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs") pod "network-metrics-daemon-2ljbv" (UID: "3c12d6fa-daae-4fc8-9325-1ae4a62765e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:13.790039 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:13.790014    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:13.790181 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:13.790168    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret podName:5c14500d-ea5d-4433-ac3c-2df82c0ef982 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:15.790138172 +0000 UTC m=+6.099065996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret") pod "global-pull-secret-syncer-hrq9b" (UID: "5c14500d-ea5d-4433-ac3c-2df82c0ef982") : object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:13.890837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:13.890085    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:13.890837 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:13.890323    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 00:36:13.890837 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:13.890347    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 00:36:13.890837 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:13.890360    2586 projected.go:196] Error preparing data for projected volume kube-api-access-blj2s for pod openshift-network-diagnostics/network-check-target-m2vg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:13.890837 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:13.890423    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s podName:4b367dde-86b2-4108-9650-90d1d9b8e47a nodeName:}" failed. No retries permitted until 2026-07-29 00:36:15.890403364 +0000 UTC m=+6.199331185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-blj2s" (UniqueName: "kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s") pod "network-check-target-m2vg8" (UID: "4b367dde-86b2-4108-9650-90d1d9b8e47a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:14.118328 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.118191    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 00:31:12 +0000 UTC" deadline="2028-04-23 05:10:15.182204882 +0000 UTC"
Jul 29 00:36:14.118328 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.118259    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15220h34m1.063949669s"
Jul 29 00:36:14.217351 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.216838    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:14.217351 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:14.216980    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:14.217351 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.217335    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:14.217653 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:14.217433    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:14.217944 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.217797    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:14.217944 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:14.217900    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:14.247835 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.247745    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-krk2j" event={"ID":"a115e159-a003-4bdc-8e43-94ac470b5d1a","Type":"ContainerStarted","Data":"c392681dfebe627930cb890af4f43c6cc44d3cdc017db3ab354b55cce9950286"}
Jul 29 00:36:14.254439 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.254398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal" event={"ID":"8f053b035387553a0f709b762fddad9a","Type":"ContainerStarted","Data":"fccf789fbc77a486f0d2c2ace9bae21b1eb18a4f7a62ddfab3ecac8c99501e03"}
Jul 29 00:36:14.257453 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.256822    2586 generic.go:356] "Generic (PLEG): container finished" podID="5807e7c8a4e7f09ca39bb57b7b6d746e" containerID="dadf78dc9ed681a37a31273f3cdf8a5bcb45821ca6e5cc85c21f3e777ea29170" exitCode=0
Jul 29 00:36:14.257453 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.256906    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal" event={"ID":"5807e7c8a4e7f09ca39bb57b7b6d746e","Type":"ContainerDied","Data":"dadf78dc9ed681a37a31273f3cdf8a5bcb45821ca6e5cc85c21f3e777ea29170"}
Jul 29 00:36:14.259539 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.259469    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-74znh" event={"ID":"e7a2ae25-317e-403b-a257-b993fe994cf2","Type":"ContainerStarted","Data":"01b130a9ccd4a6acacee21f4f9c791f66bf1b68570ade42e29812723484ce510"}
Jul 29 00:36:14.262343 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.262289    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jchqv" event={"ID":"f766f703-d84b-4ae3-9e3a-c53ede205ab8","Type":"ContainerStarted","Data":"4aeddcbd2289f95ee0aa067f90365c48d6c35bad70572b1f70fcc11a7427a6e1"}
Jul 29 00:36:14.264280 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.264155    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" event={"ID":"b06e3b5d-f144-4590-ad28-80c6a76acc5e","Type":"ContainerStarted","Data":"a95811200327abcf14135dc34150023c0a937343b1db2ffb9c229caae92be0df"}
Jul 29 00:36:14.268737 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.268683    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zg9n6" event={"ID":"f1bad3c0-d369-4188-b5d2-12090d1718e6","Type":"ContainerStarted","Data":"6fef2e653bf51bacf9d591759539a95047c93043210bf5abcf9fbf4c4c7c31de"}
Jul 29 00:36:14.270733 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:14.270118    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-227.ec2.internal" podStartSLOduration=3.270099258 podStartE2EDuration="3.270099258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:14.269174887 +0000 UTC m=+4.578102730" watchObservedRunningTime="2026-07-29 00:36:14.270099258 +0000 UTC m=+4.579027101"
Jul 29 00:36:15.805448 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:15.805404    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:15.805932 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:15.805482    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:15.805932 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:15.805657    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:15.805932 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:15.805721    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs podName:3c12d6fa-daae-4fc8-9325-1ae4a62765e9 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:19.805703063 +0000 UTC m=+10.114630887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs") pod "network-metrics-daemon-2ljbv" (UID: "3c12d6fa-daae-4fc8-9325-1ae4a62765e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:15.806144 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:15.806124    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:15.806228 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:15.806210    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret podName:5c14500d-ea5d-4433-ac3c-2df82c0ef982 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:19.806179885 +0000 UTC m=+10.115107707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret") pod "global-pull-secret-syncer-hrq9b" (UID: "5c14500d-ea5d-4433-ac3c-2df82c0ef982") : object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:15.906388 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:15.906352    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:15.906616 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:15.906597    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 00:36:15.906681 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:15.906623    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 00:36:15.906681 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:15.906637    2586 projected.go:196] Error preparing data for projected volume kube-api-access-blj2s for pod openshift-network-diagnostics/network-check-target-m2vg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:15.906791 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:15.906692    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s podName:4b367dde-86b2-4108-9650-90d1d9b8e47a nodeName:}" failed. No retries permitted until 2026-07-29 00:36:19.906673327 +0000 UTC m=+10.215601168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-blj2s" (UniqueName: "kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s") pod "network-check-target-m2vg8" (UID: "4b367dde-86b2-4108-9650-90d1d9b8e47a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:16.215984 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:16.215944    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:16.216165 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:16.215944    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:16.216165 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:16.216140    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:16.216312 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:16.215963    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:16.216312 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:16.216218    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:16.216430 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:16.216375    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:18.216507 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:18.216425    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:18.216951 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:18.216437    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:18.216951 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:18.216564    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:18.216951 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:18.216646    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:18.216951 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:18.216659    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:18.216951 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:18.216741    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:19.842419 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:19.842388    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:19.842847 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:19.842443    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:19.842847 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:19.842548    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:19.842847 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:19.842577    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:19.842847 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:19.842621    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret podName:5c14500d-ea5d-4433-ac3c-2df82c0ef982 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:27.842600433 +0000 UTC m=+18.151528270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret") pod "global-pull-secret-syncer-hrq9b" (UID: "5c14500d-ea5d-4433-ac3c-2df82c0ef982") : object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:19.842847 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:19.842641    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs podName:3c12d6fa-daae-4fc8-9325-1ae4a62765e9 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:27.842630972 +0000 UTC m=+18.151558793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs") pod "network-metrics-daemon-2ljbv" (UID: "3c12d6fa-daae-4fc8-9325-1ae4a62765e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:19.942834 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:19.942797    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:19.943019 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:19.942996    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 00:36:19.943019 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:19.943018    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 00:36:19.943140 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:19.943031    2586 projected.go:196] Error preparing data for projected volume kube-api-access-blj2s for pod openshift-network-diagnostics/network-check-target-m2vg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:19.943140 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:19.943088    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s podName:4b367dde-86b2-4108-9650-90d1d9b8e47a nodeName:}" failed. No retries permitted until 2026-07-29 00:36:27.943069448 +0000 UTC m=+18.251997269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-blj2s" (UniqueName: "kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s") pod "network-check-target-m2vg8" (UID: "4b367dde-86b2-4108-9650-90d1d9b8e47a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:20.217228 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:20.216705    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:20.217228 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:20.216825    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:20.217228 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:20.216839    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:20.217228 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:20.216881    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:20.217228 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:20.216972    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:20.217228 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:20.217047    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:22.215410 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:22.215378    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:22.215827 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:22.215395    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:22.215827 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:22.215505    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:22.215827 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:22.215508    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:22.215827 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:22.215609    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:22.215827 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:22.215686    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:24.216328 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:24.216232    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:24.216328 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:24.216276    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:24.216328 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:24.216233    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:24.216879 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:24.216363    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:24.216879 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:24.216486    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:24.216879 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:24.216578    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:26.215565 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:26.215527    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:26.216024 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:26.215537    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:26.216024 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:26.215667    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:26.216024 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:26.215539    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:26.216024 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:26.215762    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:26.216024 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:26.215835    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:27.902407 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:27.902376    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:27.902927 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:27.902421    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:27.902927 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:27.902533    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:27.902927 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:27.902589    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret podName:5c14500d-ea5d-4433-ac3c-2df82c0ef982 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:43.902572885 +0000 UTC m=+34.211500705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret") pod "global-pull-secret-syncer-hrq9b" (UID: "5c14500d-ea5d-4433-ac3c-2df82c0ef982") : object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:27.902927 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:27.902541    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:27.902927 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:27.902623    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs podName:3c12d6fa-daae-4fc8-9325-1ae4a62765e9 nodeName:}" failed. No retries permitted until 2026-07-29 00:36:43.902613258 +0000 UTC m=+34.211541084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs") pod "network-metrics-daemon-2ljbv" (UID: "3c12d6fa-daae-4fc8-9325-1ae4a62765e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:28.003773 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:28.003726    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:28.003960 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:28.003887    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 00:36:28.003960 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:28.003917    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 00:36:28.003960 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:28.003932    2586 projected.go:196] Error preparing data for projected volume kube-api-access-blj2s for pod openshift-network-diagnostics/network-check-target-m2vg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:28.004108 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:28.003999    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s podName:4b367dde-86b2-4108-9650-90d1d9b8e47a nodeName:}" failed. No retries permitted until 2026-07-29 00:36:44.003978058 +0000 UTC m=+34.312905879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-blj2s" (UniqueName: "kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s") pod "network-check-target-m2vg8" (UID: "4b367dde-86b2-4108-9650-90d1d9b8e47a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:28.215520 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:28.215428    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:28.215520 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:28.215427    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:28.215719 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:28.215436    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:28.215719 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:28.215583    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:28.215719 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:28.215663    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:28.215843 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:28.215775    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:30.216932 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:30.216901    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:30.217438 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:30.216986    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:30.217438 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:30.217009    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:30.217438 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:30.217154    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:30.217438 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:30.217188    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:30.217438 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:30.217288    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:32.215831 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.215387    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:32.216782 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.215386    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:32.216782 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.215430    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:32.216782 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:32.215955    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:32.216782 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:32.216120    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:32.216782 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:32.216241    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:32.308232 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.307881    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" event={"ID":"b06e3b5d-f144-4590-ad28-80c6a76acc5e","Type":"ContainerStarted","Data":"3e28510bc7d917be7c1f6422a4160d0460eb741d1716bb0658d152a1c43f61dd"}
Jul 29 00:36:32.308232 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.307970    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" event={"ID":"b06e3b5d-f144-4590-ad28-80c6a76acc5e","Type":"ContainerStarted","Data":"c89b46179d6faa663ceab5cf1b1494cecf7a0a594d7cbe9f0030a16e04af30a7"}
Jul 29 00:36:32.308232 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.308013    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" event={"ID":"b06e3b5d-f144-4590-ad28-80c6a76acc5e","Type":"ContainerStarted","Data":"0760cd744e6d99d517f5ccb60da086647feb19dfe44eb79e525c3e82188a1627"}
Jul 29 00:36:32.308232 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.308058    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" event={"ID":"b06e3b5d-f144-4590-ad28-80c6a76acc5e","Type":"ContainerStarted","Data":"d4eb0daa13b556a57af3c5a261cef7da59d84066bba47d1de082246dcbdfb7fd"}
Jul 29 00:36:32.308232 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.308101    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" event={"ID":"b06e3b5d-f144-4590-ad28-80c6a76acc5e","Type":"ContainerStarted","Data":"21464ea119eeb2982f0dcc4ec07ed377e83f8b474992b6387c4b0bb804b98a2a"}
Jul 29 00:36:32.308232 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.308146    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" event={"ID":"b06e3b5d-f144-4590-ad28-80c6a76acc5e","Type":"ContainerStarted","Data":"42c7b6b81f2be40159c267265312246935c0eedb5cb19b181c720cd6a2ce2baf"}
Jul 29 00:36:32.310419 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.310388    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr" event={"ID":"c308d7f1-f1da-4328-895a-e3790e8ca866","Type":"ContainerStarted","Data":"9aa3f59250575942e37f35a86b52365765a1f321a95d312875862b5914a78c56"}
Jul 29 00:36:32.311852 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.311827    2586 generic.go:356] "Generic (PLEG): container finished" podID="4442676d-6797-4d73-bdde-af2342f528e5" containerID="44e67a7f949f4f0d01302aab7b31235e4cc535cfd9dbd52195f401ae5193fdc3" exitCode=0
Jul 29 00:36:32.312010 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.311893    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbnr9" event={"ID":"4442676d-6797-4d73-bdde-af2342f528e5","Type":"ContainerDied","Data":"44e67a7f949f4f0d01302aab7b31235e4cc535cfd9dbd52195f401ae5193fdc3"}
Jul 29 00:36:32.313606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.313457    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2t4gv" event={"ID":"acbd27bf-81aa-49e9-9c59-d58bcf765cd8","Type":"ContainerStarted","Data":"323a71713a7a14c881b580be87407f897ad19dcbaafd8510c0472fbe4e4f2521"}
Jul 29 00:36:32.314919 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.314897    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-krk2j" event={"ID":"a115e159-a003-4bdc-8e43-94ac470b5d1a","Type":"ContainerStarted","Data":"49cd5a678ed66cf5fa2c5221426b8eee32f653933629e8f0dc579acfd342434d"}
Jul 29 00:36:32.316641 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.316580    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal" event={"ID":"5807e7c8a4e7f09ca39bb57b7b6d746e","Type":"ContainerStarted","Data":"068c9d24f2c23fba97868a0203d908ab7baa9c347a7a45c0a49c21f2e07d5465"}
Jul 29 00:36:32.317947 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.317911    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-57bk9" event={"ID":"f5536843-3e0d-4cbb-bfa8-938694ab3e63","Type":"ContainerStarted","Data":"a6d9474f56ebd99bcae1e10897b44c3baf38aaf1353db13e33a34335f6244477"}
Jul 29 00:36:32.319687 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.319668    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-74znh" event={"ID":"e7a2ae25-317e-403b-a257-b993fe994cf2","Type":"ContainerStarted","Data":"44bf35b161c03e31ac133adb6561cd2a5e323cf289c9894f57103d6135113079"}
Jul 29 00:36:32.321072 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.321052    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jchqv" event={"ID":"f766f703-d84b-4ae3-9e3a-c53ede205ab8","Type":"ContainerStarted","Data":"a198cad993567ff6dfb40ca4813834b253c4addcf40089694c23811bd0cd3f5c"}
Jul 29 00:36:32.355089 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.355035    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jchqv" podStartSLOduration=4.7877820490000005 podStartE2EDuration="22.355019965s" totalImagesPullingTime="17.567237916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:32.354815048 +0000 UTC m=+22.663742907" watchObservedRunningTime="2026-07-29 00:36:32.355019965 +0000 UTC m=+22.663947806"
Jul 29 00:36:32.370048 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.370006    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-227.ec2.internal" podStartSLOduration=21.369991283 podStartE2EDuration="21.369991283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:32.369485831 +0000 UTC m=+22.678413674" watchObservedRunningTime="2026-07-29 00:36:32.369991283 +0000 UTC m=+22.678919125"
Jul 29 00:36:32.388162 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.388119    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-57bk9" podStartSLOduration=3.212149887 podStartE2EDuration="22.388104682s" totalImagesPullingTime="19.175954795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:32.387846338 +0000 UTC m=+22.696774180" watchObservedRunningTime="2026-07-29 00:36:32.388104682 +0000 UTC m=+22.697032523"
Jul 29 00:36:32.403288 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.403245    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2t4gv" podStartSLOduration=3.519219404 podStartE2EDuration="22.40322947s" totalImagesPullingTime="18.884010066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:32.403069771 +0000 UTC m=+22.711997613" watchObservedRunningTime="2026-07-29 00:36:32.40322947 +0000 UTC m=+22.712157313"
Jul 29 00:36:32.441002 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.440951    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-krk2j" podStartSLOduration=4.872612694 podStartE2EDuration="22.440936545s" totalImagesPullingTime="17.568323851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:32.440665874 +0000 UTC m=+22.749593717" watchObservedRunningTime="2026-07-29 00:36:32.440936545 +0000 UTC m=+22.749864386"
Jul 29 00:36:32.441186 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.441029    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-74znh" podStartSLOduration=4.906601396 podStartE2EDuration="22.441024239s" totalImagesPullingTime="17.534422843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:32.422413479 +0000 UTC m=+22.731341321" watchObservedRunningTime="2026-07-29 00:36:32.441024239 +0000 UTC m=+22.749952082"
Jul 29 00:36:32.531768 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.531729    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:32.591713 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.591637    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:32.691812 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.691785    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:32.752454 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.752426    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:32.965842 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:32.965783    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 00:36:33.133190 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.133050    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T00:36:32.965811078Z","UUID":"e6a335fd-8c81-4cf7-a71a-e009072ed1de","Handler":null,"Name":"","Endpoint":""}
Jul 29 00:36:33.134809 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.134777    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 00:36:33.134949 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.134819    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 00:36:33.325904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.325864    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zg9n6" event={"ID":"f1bad3c0-d369-4188-b5d2-12090d1718e6","Type":"ContainerStarted","Data":"4beaa61bcd0d7b9e545d5657e280ce58dc1f85339e5620041c8f46b045dbd3da"}
Jul 29 00:36:33.327932 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.327906    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr" event={"ID":"c308d7f1-f1da-4328-895a-e3790e8ca866","Type":"ContainerStarted","Data":"000f52f5a0aa969604bf749e8d4969ce03503f2ad69c32fc0cc6c071c02e03f4"}
Jul 29 00:36:33.341369 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.341315    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zg9n6" podStartSLOduration=5.77228498 podStartE2EDuration="23.341300221s" totalImagesPullingTime="17.569015241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:33.341018454 +0000 UTC m=+23.649946296" watchObservedRunningTime="2026-07-29 00:36:33.341300221 +0000 UTC m=+23.650228063"
Jul 29 00:36:33.386274 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.386166    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:33.386869 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.386824    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:33.461440 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.461371    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:33.560879 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:33.560844    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:34.215979 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:34.215904    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:34.216126 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:34.215904    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:34.216126 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:34.216030    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:34.216126 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:34.216041    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:34.216290 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:34.216136    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:34.216290 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:34.216257    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:34.299428 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:34.299391    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:34.334402 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:34.334365    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" event={"ID":"b06e3b5d-f144-4590-ad28-80c6a76acc5e","Type":"ContainerStarted","Data":"ddeb182f49d7fdc0c2c1c9da6c6342393a748dafffd13a07cda2d660c927338e"}
Jul 29 00:36:34.336652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:34.336625    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr" event={"ID":"c308d7f1-f1da-4328-895a-e3790e8ca866","Type":"ContainerStarted","Data":"d0bac05e3320d131442554ce1d94fc3e283baefb86a93d85648b9cf60007d164"}
Jul 29 00:36:34.354493 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:34.354461    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:34.355568 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:34.355524    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s6cr" podStartSLOduration=3.213065245 podStartE2EDuration="24.355505641s" totalImagesPullingTime="21.142440396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:34.354898957 +0000 UTC m=+24.663826826" watchObservedRunningTime="2026-07-29 00:36:34.355505641 +0000 UTC m=+24.664433483"
Jul 29 00:36:35.257004 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:35.256974    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:35.339070 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:35.339039    2586 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 00:36:36.216446 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:36.216252    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:36.216613 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:36.216252    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:36.216613 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:36.216538    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:36.216729 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:36.216623    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:36.216729 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:36.216300    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:36.216812 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:36.216747    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:36.342354 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:36.342323    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:36.509436 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:36.509214    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:36.581339 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:36.580997    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:36.641432 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:36.641410    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:36.701639 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:36.701608    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:36.802212 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:36.802172    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:37.124920 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:37.124888    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:37.345086 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:37.345057    2586 generic.go:356] "Generic (PLEG): container finished" podID="4442676d-6797-4d73-bdde-af2342f528e5" containerID="12739f84589f1a9fb5f975ba9b545722f00abd1e309395df8f3bec9e488b4488" exitCode=0
Jul 29 00:36:37.345887 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:37.345145    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbnr9" event={"ID":"4442676d-6797-4d73-bdde-af2342f528e5","Type":"ContainerDied","Data":"12739f84589f1a9fb5f975ba9b545722f00abd1e309395df8f3bec9e488b4488"}
Jul 29 00:36:37.348753 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:37.348727    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" event={"ID":"b06e3b5d-f144-4590-ad28-80c6a76acc5e","Type":"ContainerStarted","Data":"0762ec01f8949d0330b1259db9f3035e15f868255a8accb7ccdccb5531ee90e6"}
Jul 29 00:36:37.348977 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:37.348962    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:37.349023 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:37.349016    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:37.363930 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:37.363802    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:37.395429 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:37.395359    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq" podStartSLOduration=9.80697029 podStartE2EDuration="27.395346785s" totalImagesPullingTime="17.588376495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:37.393970216 +0000 UTC m=+27.702898057" watchObservedRunningTime="2026-07-29 00:36:37.395346785 +0000 UTC m=+27.704274624"
Jul 29 00:36:37.645085 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:37.645056    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:38.029149 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:38.029099    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:38.103513 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:38.103483    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:38.213129 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:38.213098    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:38.215421 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:38.215401    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:38.215531 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:38.215401    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:38.215531 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:38.215517    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:38.215605 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:38.215526    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:38.215650 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:38.215627    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:38.215732 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:38.215714    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:38.350929 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:38.350902    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:38.365999 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:38.365973    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:36:38.834001 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:38.833970    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:39.029294 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:39.029189    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:39.354633 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:39.354600    2586 generic.go:356] "Generic (PLEG): container finished" podID="4442676d-6797-4d73-bdde-af2342f528e5" containerID="ca3636d1333f2ec6ebb949eb41858e9a8a9025934460790546d05d2e90fc6662" exitCode=0
Jul 29 00:36:39.354976 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:39.354679    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbnr9" event={"ID":"4442676d-6797-4d73-bdde-af2342f528e5","Type":"ContainerDied","Data":"ca3636d1333f2ec6ebb949eb41858e9a8a9025934460790546d05d2e90fc6662"}
Jul 29 00:36:39.469795 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:39.469761    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:39.630966 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:39.630882    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:39.826926 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:39.826899    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:40.029841 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.029764    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:40.183310 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.183277    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:40.216795 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.216770    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:40.216976 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:40.216863    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:40.216976 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.216873    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:40.216976 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.216903    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:40.216976 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:40.216946    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:40.217207 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:40.217023    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:40.501127 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.501098    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:40.501578 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.501242    2586 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 00:36:40.501708 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.501690    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2t4gv"
Jul 29 00:36:40.828760 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.828384    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:40.928798 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:40.928766    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:41.027932 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:41.027846    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:41.227352 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:41.227319    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:41.361577 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:41.361542    2586 generic.go:356] "Generic (PLEG): container finished" podID="4442676d-6797-4d73-bdde-af2342f528e5" containerID="b86e4532f3df4753d57d7813512f3f7a4872b31dba916cacf298662833a7b067" exitCode=0
Jul 29 00:36:41.361757 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:41.361627    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbnr9" event={"ID":"4442676d-6797-4d73-bdde-af2342f528e5","Type":"ContainerDied","Data":"b86e4532f3df4753d57d7813512f3f7a4872b31dba916cacf298662833a7b067"}
Jul 29 00:36:41.423615 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:41.423582    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:41.689241 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:41.689112    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:42.216187 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:42.216153    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:42.216187 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:42.216186    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:42.216479 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:42.216153    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:42.216479 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:42.216308    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:42.216479 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:42.216368    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:42.216479 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:42.216443    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:42.444606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:42.444571    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:42.545964 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:42.545840    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:42.829910 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:42.829868    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:43.027974 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:43.027939    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:43.227220 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:43.227108    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:43.625683 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:43.625650    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:43.927024 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:43.926926    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:43.927024 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:43.926998    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:43.927509 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:43.927061    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:43.927509 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:43.927116    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:43.927509 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:43.927137    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret podName:5c14500d-ea5d-4433-ac3c-2df82c0ef982 nodeName:}" failed. No retries permitted until 2026-07-29 00:37:15.927115491 +0000 UTC m=+66.236043317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret") pod "global-pull-secret-syncer-hrq9b" (UID: "5c14500d-ea5d-4433-ac3c-2df82c0ef982") : object "kube-system"/"original-pull-secret" not registered
Jul 29 00:36:43.927509 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:43.927161    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs podName:3c12d6fa-daae-4fc8-9325-1ae4a62765e9 nodeName:}" failed. No retries permitted until 2026-07-29 00:37:15.92714499 +0000 UTC m=+66.236072828 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs") pod "network-metrics-daemon-2ljbv" (UID: "3c12d6fa-daae-4fc8-9325-1ae4a62765e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 00:36:44.027534 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:44.027496    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:44.027723 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:44.027634    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 00:36:44.027723 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:44.027656    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 00:36:44.027723 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:44.027669    2586 projected.go:196] Error preparing data for projected volume kube-api-access-blj2s for pod openshift-network-diagnostics/network-check-target-m2vg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:44.027882 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:44.027731    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s podName:4b367dde-86b2-4108-9650-90d1d9b8e47a nodeName:}" failed. No retries permitted until 2026-07-29 00:37:16.027713065 +0000 UTC m=+66.336640899 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-blj2s" (UniqueName: "kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s") pod "network-check-target-m2vg8" (UID: "4b367dde-86b2-4108-9650-90d1d9b8e47a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 00:36:44.032102 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:44.032079    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:44.215485 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:44.215393    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:44.215485 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:44.215420    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:44.215665 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:44.215514    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:44.215665 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:44.215533    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:44.215665 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:44.215608    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:44.215771 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:44.215665    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:44.641986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:44.641955    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:45.025522 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:45.025434    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:45.428003 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:45.427978    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:45.626411 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:45.626381    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:46.215306 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.215268    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:46.215306 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.215292    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:46.215825 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.215375    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:46.215825 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:46.215408    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:46.215825 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:46.215420    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:46.215825 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:46.215496    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:46.230867 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.230841    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:46.376892 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.376861    2586 generic.go:356] "Generic (PLEG): container finished" podID="4442676d-6797-4d73-bdde-af2342f528e5" containerID="6d97f2dc4d24a74456d60c8f01b92909f61b534aa03e47f9f238d129d36fa903" exitCode=0
Jul 29 00:36:46.377074 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.376900    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbnr9" event={"ID":"4442676d-6797-4d73-bdde-af2342f528e5","Type":"ContainerDied","Data":"6d97f2dc4d24a74456d60c8f01b92909f61b534aa03e47f9f238d129d36fa903"}
Jul 29 00:36:46.445365 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.445335    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:46.578631 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.578595    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:46.679439 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.679403    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:46.736135 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.736051    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:46.913487 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:46.913456    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:47.059937 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:47.059910    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:47.161973 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:47.161942    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:47.382329 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:47.382252    2586 generic.go:356] "Generic (PLEG): container finished" podID="4442676d-6797-4d73-bdde-af2342f528e5" containerID="4a67a7d30ae4323893b9617972f049509a9b3f67416744e6aa29cacd39446d63" exitCode=0
Jul 29 00:36:47.382329 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:47.382290    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbnr9" event={"ID":"4442676d-6797-4d73-bdde-af2342f528e5","Type":"ContainerDied","Data":"4a67a7d30ae4323893b9617972f049509a9b3f67416744e6aa29cacd39446d63"}
Jul 29 00:36:47.486733 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:47.486698    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:47.553019 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:47.552986    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:47.674865 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:47.674778    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:47.882778 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:47.882749    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:47.933673 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:47.933591    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:48.216042 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:48.215958    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:48.216187 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:48.215958    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:48.216187 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:48.216075    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:48.216187 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:48.216163    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:48.216335 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:48.215958    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:48.216335 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:48.216287    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:48.259057 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:48.259027    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:48.387699 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:48.387667    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbnr9" event={"ID":"4442676d-6797-4d73-bdde-af2342f528e5","Type":"ContainerStarted","Data":"d2b31f1bb8e639df18f7cfcf7a08d15ffec7410a2a896c5b57ea8374fc313fcb"}
Jul 29 00:36:48.413542 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:48.413492    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bbnr9" podStartSLOduration=8.660416203 podStartE2EDuration="38.413473639s" totalImagesPullingTime="29.753057436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:36:48.411599533 +0000 UTC m=+38.720527376" watchObservedRunningTime="2026-07-29 00:36:48.413473639 +0000 UTC m=+38.722401482"
Jul 29 00:36:48.474175 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:48.474096    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:48.958239 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:48.958190    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:49.276898 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:49.276816    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:49.377929 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:49.377893    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:50.217013 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:50.216984    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:50.217474 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:50.217064    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:50.217474 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:50.217100    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:50.217474 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:50.217189    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:50.217474 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:50.217100    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:50.217474 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:50.217343    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:52.216284 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:52.216247    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:52.216284 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:52.216285    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:52.216756 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:52.216359    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:52.216756 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:52.216380    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:52.216756 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:52.216495    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:52.216756 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:52.216542    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:53.680046 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:53.680013    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:53.768867 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:53.768831    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:54.215826 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:54.215803    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:54.216128 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:54.215803    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:54.216128 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:54.215894    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:54.216128 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:54.215989    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:54.216128 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:54.216048    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:54.216128 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:54.216101    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:56.216010 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:56.215979    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:56.216572 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:56.215979    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:56.216572 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:56.216080    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:56.216572 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:56.215989    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:56.216572 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:56.216146    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:56.216572 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:56.216247    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:58.215805 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:58.215766    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:58.215805 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:58.215792    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:58.216251 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:58.215766    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:58.216251 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:58.215893    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:58.216251 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:58.215944    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:58.216251 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:58.216025    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:36:58.604830 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:58.604789    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:58.748318 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:58.748278    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:58.841904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:58.841863    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:58.936742 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:58.936655    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:59.036408 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:59.036380    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:36:59.653480 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:59.653247    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrq9b"]
Jul 29 00:36:59.654275 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:59.654132    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:36:59.654355 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:59.654276    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:36:59.657783 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:59.657745    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2vg8"]
Jul 29 00:36:59.658597 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:59.658010    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:36:59.658597 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:59.658128    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:36:59.658597 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:59.658490    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2ljbv"]
Jul 29 00:36:59.658597 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:36:59.658579    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:36:59.658866 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:36:59.658681    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:37:00.126956 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:00.126925    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:00.219024 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:00.218996    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:01.216088 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:01.216052    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:37:01.216606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:01.216151    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:37:01.216606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:01.216161    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:37:01.216606 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:01.216152    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:37:01.216606 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:01.216269    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:37:01.216606 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:01.216323    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:37:03.115524 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:03.115492    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:03.182401 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:03.182356    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:03.215368 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:03.215333    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:37:03.215529 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:03.215334    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:37:03.215529 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:03.215334    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:37:03.215529 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:03.215467    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ljbv" podUID="3c12d6fa-daae-4fc8-9325-1ae4a62765e9"
Jul 29 00:37:03.215682 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:03.215541    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2vg8" podUID="4b367dde-86b2-4108-9650-90d1d9b8e47a"
Jul 29 00:37:03.215682 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:03.215631    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrq9b" podUID="5c14500d-ea5d-4433-ac3c-2df82c0ef982"
Jul 29 00:37:04.028432 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.028405    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-227.ec2.internal" event="NodeReady"
Jul 29 00:37:04.028562 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.028541    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 00:37:04.078519 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.078486    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sqrrw"]
Jul 29 00:37:04.080426 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.080399    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.080551 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.080443    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5cv9f"]
Jul 29 00:37:04.082356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.082337    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5cv9f"
Jul 29 00:37:04.083047 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.083031    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2nxv8\""
Jul 29 00:37:04.083129 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.083031    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 00:37:04.084311 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.084294    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 00:37:04.085966 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.085949    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 00:37:04.086037 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.086004    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 00:37:04.086652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.086637    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 00:37:04.086941 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.086923    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5x4gq\""
Jul 29 00:37:04.094329 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.094307    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqrrw"]
Jul 29 00:37:04.094959 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.094940    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5cv9f"]
Jul 29 00:37:04.207813 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.207703    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:04.265547 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.265511    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-tmp-dir\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.265547 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.265547    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6g6\" (UniqueName: \"kubernetes.io/projected/c12e1869-ccfe-438d-838c-e960413a9c1e-kube-api-access-lh6g6\") pod \"ingress-canary-5cv9f\" (UID: \"c12e1869-ccfe-438d-838c-e960413a9c1e\") " pod="openshift-ingress-canary/ingress-canary-5cv9f"
Jul 29 00:37:04.265791 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.265642    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-config-volume\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.265791 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.265678    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-metrics-tls\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.265791 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.265706    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8664v\" (UniqueName: \"kubernetes.io/projected/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-kube-api-access-8664v\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.265791 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.265761    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c12e1869-ccfe-438d-838c-e960413a9c1e-cert\") pod \"ingress-canary-5cv9f\" (UID: \"c12e1869-ccfe-438d-838c-e960413a9c1e\") " pod="openshift-ingress-canary/ingress-canary-5cv9f"
Jul 29 00:37:04.277535 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.277507    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:04.367099 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.367067    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-config-volume\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.367099 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.367101    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-metrics-tls\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.367321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.367126    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8664v\" (UniqueName: \"kubernetes.io/projected/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-kube-api-access-8664v\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.367321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.367159    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c12e1869-ccfe-438d-838c-e960413a9c1e-cert\") pod \"ingress-canary-5cv9f\" (UID: \"c12e1869-ccfe-438d-838c-e960413a9c1e\") " pod="openshift-ingress-canary/ingress-canary-5cv9f"
Jul 29 00:37:04.367321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.367224    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-tmp-dir\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.367321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.367251    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6g6\" (UniqueName: \"kubernetes.io/projected/c12e1869-ccfe-438d-838c-e960413a9c1e-kube-api-access-lh6g6\") pod \"ingress-canary-5cv9f\" (UID: \"c12e1869-ccfe-438d-838c-e960413a9c1e\") " pod="openshift-ingress-canary/ingress-canary-5cv9f"
Jul 29 00:37:04.367652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.367632    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-tmp-dir\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.367741 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.367724    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-config-volume\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.371402 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.371275    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-metrics-tls\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.371500 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.371313    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c12e1869-ccfe-438d-838c-e960413a9c1e-cert\") pod \"ingress-canary-5cv9f\" (UID: \"c12e1869-ccfe-438d-838c-e960413a9c1e\") " pod="openshift-ingress-canary/ingress-canary-5cv9f"
Jul 29 00:37:04.376514 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.376483    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8664v\" (UniqueName: \"kubernetes.io/projected/b5e0eb10-33c8-4f4a-9076-f3ccda11708c-kube-api-access-8664v\") pod \"dns-default-sqrrw\" (UID: \"b5e0eb10-33c8-4f4a-9076-f3ccda11708c\") " pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.376632 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.376539    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6g6\" (UniqueName: \"kubernetes.io/projected/c12e1869-ccfe-438d-838c-e960413a9c1e-kube-api-access-lh6g6\") pod \"ingress-canary-5cv9f\" (UID: \"c12e1869-ccfe-438d-838c-e960413a9c1e\") " pod="openshift-ingress-canary/ingress-canary-5cv9f"
Jul 29 00:37:04.391400 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.391372    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:04.396634 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.396617    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5cv9f"
Jul 29 00:37:04.504276 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.504245    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84c48b94b9-cbwlb"]
Jul 29 00:37:04.507762 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.506734    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.510473 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.510090    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 00:37:04.510473 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.510307    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 00:37:04.510473 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.510369    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 00:37:04.510473 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.510397    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 00:37:04.511523 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.511332    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 00:37:04.511639 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.511576    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 00:37:04.511791 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.511775    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nlj79\""
Jul 29 00:37:04.512399 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.512286    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 00:37:04.524356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.524161    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c48b94b9-cbwlb"]
Jul 29 00:37:04.552006 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.551977    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5cv9f"]
Jul 29 00:37:04.556365 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:04.556337    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12e1869_ccfe_438d_838c_e960413a9c1e.slice/crio-5bb74fdbe89b4b119e1691b3c852db8e2f8b5e1b0fffd6c693096724a7771548 WatchSource:0}: Error finding container 5bb74fdbe89b4b119e1691b3c852db8e2f8b5e1b0fffd6c693096724a7771548: Status 404 returned error can't find the container with id 5bb74fdbe89b4b119e1691b3c852db8e2f8b5e1b0fffd6c693096724a7771548
Jul 29 00:37:04.558627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.558586    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqrrw"]
Jul 29 00:37:04.563389 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:04.563349    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e0eb10_33c8_4f4a_9076_f3ccda11708c.slice/crio-a1d1cd1f12e7c6a1a5f42da185145fb86767fb77e5e87a718057051c9c78921a WatchSource:0}: Error finding container a1d1cd1f12e7c6a1a5f42da185145fb86767fb77e5e87a718057051c9c78921a: Status 404 returned error can't find the container with id a1d1cd1f12e7c6a1a5f42da185145fb86767fb77e5e87a718057051c9c78921a
Jul 29 00:37:04.598365 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.598328    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:04.669547 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.669508    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-config\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.669715 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.669568    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-oauth-serving-cert\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.669715 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.669590    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-serving-cert\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.669715 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.669615    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-oauth-config\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.669715 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.669683    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-service-ca\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.669846 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.669725    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7qf\" (UniqueName: \"kubernetes.io/projected/a781e516-69f2-4186-a7c1-3af8465aeb0d-kube-api-access-6z7qf\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.671478 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.671458    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:04.770789 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.770702    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-serving-cert\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.770789 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.770747    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-oauth-config\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.770974 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.770798    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-service-ca\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.770974 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.770829    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7qf\" (UniqueName: \"kubernetes.io/projected/a781e516-69f2-4186-a7c1-3af8465aeb0d-kube-api-access-6z7qf\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.771066 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.771036    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-config\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.771142 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.771125    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-oauth-serving-cert\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.771765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.771665    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-config\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.771765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.771710    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-service-ca\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.771765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.771739    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-oauth-serving-cert\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.773832 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.773815    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-oauth-config\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.773889 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.773863    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-serving-cert\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.779406 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.779387    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7qf\" (UniqueName: \"kubernetes.io/projected/a781e516-69f2-4186-a7c1-3af8465aeb0d-kube-api-access-6z7qf\") pod \"console-84c48b94b9-cbwlb\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") " pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.825909 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.825879    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:04.971532 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:04.971498    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c48b94b9-cbwlb"]
Jul 29 00:37:04.974741 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:04.974710    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda781e516_69f2_4186_a7c1_3af8465aeb0d.slice/crio-71613fd8c77331e21f173c3c7f8f414435f7349904dc9a14719c48581dd8f9ad WatchSource:0}: Error finding container 71613fd8c77331e21f173c3c7f8f414435f7349904dc9a14719c48581dd8f9ad: Status 404 returned error can't find the container with id 71613fd8c77331e21f173c3c7f8f414435f7349904dc9a14719c48581dd8f9ad
Jul 29 00:37:05.216336 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.216242    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:37:05.216336 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.216265    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:37:05.216864 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.216633    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:37:05.221243 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.221008    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 00:37:05.221243 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.221171    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4b5n\""
Jul 29 00:37:05.221486 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.221319    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 00:37:05.221486 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.221411    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-44db9\""
Jul 29 00:37:05.221595 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.221524    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 00:37:05.221692 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.221660    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 00:37:05.426378 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.426339    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c48b94b9-cbwlb" event={"ID":"a781e516-69f2-4186-a7c1-3af8465aeb0d","Type":"ContainerStarted","Data":"71613fd8c77331e21f173c3c7f8f414435f7349904dc9a14719c48581dd8f9ad"}
Jul 29 00:37:05.427938 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.427908    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqrrw" event={"ID":"b5e0eb10-33c8-4f4a-9076-f3ccda11708c","Type":"ContainerStarted","Data":"a1d1cd1f12e7c6a1a5f42da185145fb86767fb77e5e87a718057051c9c78921a"}
Jul 29 00:37:05.429479 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:05.429450    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5cv9f" event={"ID":"c12e1869-ccfe-438d-838c-e960413a9c1e","Type":"ContainerStarted","Data":"5bb74fdbe89b4b119e1691b3c852db8e2f8b5e1b0fffd6c693096724a7771548"}
Jul 29 00:37:07.297334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:07.297303    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:07.362871 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:07.362834    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:08.051423 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:08.051384    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:08.113656 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:08.113610    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:08.935938 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:08.935897    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cc88b5bd6-fkcng"]
Jul 29 00:37:08.958544 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:08.958520    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:08.965752 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:08.965724    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc88b5bd6-fkcng"]
Jul 29 00:37:08.968386 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:08.968361    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:08.969281 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:08.969260    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 00:37:09.004630 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.004594    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-oauth-serving-cert\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.004769 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.004649    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-serving-cert\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.004769 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.004727    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-oauth-config\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.004769 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.004763    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-trusted-ca-bundle\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.004877 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.004783    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-config\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.004877 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.004801    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-service-ca\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.004877 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.004821    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc68p\" (UniqueName: \"kubernetes.io/projected/81d23d92-4d3b-4ed0-98d1-d22e588537fa-kube-api-access-cc68p\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.054378 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.054342    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:09.105468 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.105428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-service-ca\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.105468 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.105470    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc68p\" (UniqueName: \"kubernetes.io/projected/81d23d92-4d3b-4ed0-98d1-d22e588537fa-kube-api-access-cc68p\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.105709 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.105498    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-oauth-serving-cert\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.105709 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.105552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-serving-cert\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.105709 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.105596    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-oauth-config\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.105709 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.105630    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-trusted-ca-bundle\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.105709 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.105652    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-config\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.106313 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.106285    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-service-ca\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.106436 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.106341    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-oauth-serving-cert\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.106500 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.106457    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-config\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.106744 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.106720    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-trusted-ca-bundle\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.108019 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.108001    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-oauth-config\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.108257 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.108234    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-serving-cert\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.119846 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.119819    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc68p\" (UniqueName: \"kubernetes.io/projected/81d23d92-4d3b-4ed0-98d1-d22e588537fa-kube-api-access-cc68p\") pod \"console-7cc88b5bd6-fkcng\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") " pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.267375 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.267339    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:09.417980 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.417947    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc88b5bd6-fkcng"]
Jul 29 00:37:09.421385 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:09.421355    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d23d92_4d3b_4ed0_98d1_d22e588537fa.slice/crio-7ee042b91fafad922d3c5b20ef1071293fa8ff86aa0f3a6b71dcacf52c4ec22a WatchSource:0}: Error finding container 7ee042b91fafad922d3c5b20ef1071293fa8ff86aa0f3a6b71dcacf52c4ec22a: Status 404 returned error can't find the container with id 7ee042b91fafad922d3c5b20ef1071293fa8ff86aa0f3a6b71dcacf52c4ec22a
Jul 29 00:37:09.421792 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.421767    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:09.439671 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.439641    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5cv9f" event={"ID":"c12e1869-ccfe-438d-838c-e960413a9c1e","Type":"ContainerStarted","Data":"1ef8b8b05e8ea345f2e0c4c294fa3cd9c0ad055903f1b00e8a9aeb8d21886f35"}
Jul 29 00:37:09.441230 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.441187    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c48b94b9-cbwlb" event={"ID":"a781e516-69f2-4186-a7c1-3af8465aeb0d","Type":"ContainerStarted","Data":"4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252"}
Jul 29 00:37:09.442876 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.442847    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqrrw" event={"ID":"b5e0eb10-33c8-4f4a-9076-f3ccda11708c","Type":"ContainerStarted","Data":"04af1ff53aa74f60524e54d41cbded8532dc8ff7ae82d33d002135217372ef68"}
Jul 29 00:37:09.442963 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.442901    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqrrw" event={"ID":"b5e0eb10-33c8-4f4a-9076-f3ccda11708c","Type":"ContainerStarted","Data":"946466e4f385210634d8a3c7f981f40ea39b38462e12e64d86b21d69833b43c0"}
Jul 29 00:37:09.444146 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.444124    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc88b5bd6-fkcng" event={"ID":"81d23d92-4d3b-4ed0-98d1-d22e588537fa","Type":"ContainerStarted","Data":"7ee042b91fafad922d3c5b20ef1071293fa8ff86aa0f3a6b71dcacf52c4ec22a"}
Jul 29 00:37:09.474120 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.474073    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5cv9f" podStartSLOduration=1.557705849 podStartE2EDuration="5.474053857s" totalImagesPullingTime="3.916348008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:09.473685564 +0000 UTC m=+59.782613405" watchObservedRunningTime="2026-07-29 00:37:09.474053857 +0000 UTC m=+59.782981698"
Jul 29 00:37:09.491612 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.491562    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84c48b94b9-cbwlb" podStartSLOduration=1.987182891 podStartE2EDuration="5.491546145s" totalImagesPullingTime="3.504363254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:09.490791739 +0000 UTC m=+59.799719591" watchObservedRunningTime="2026-07-29 00:37:09.491546145 +0000 UTC m=+59.800473968"
Jul 29 00:37:09.492223 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.492184    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:09.507781 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.507592    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sqrrw" podStartSLOduration=1.605451999 podStartE2EDuration="5.507570389s" totalImagesPullingTime="3.90211839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:09.507300067 +0000 UTC m=+59.816227910" watchObservedRunningTime="2026-07-29 00:37:09.507570389 +0000 UTC m=+59.816498232"
Jul 29 00:37:09.627859 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.627830    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:09.634309 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.634284    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"]
Jul 29 00:37:09.648712 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.648677    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"
Jul 29 00:37:09.649172 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.649145    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"]
Jul 29 00:37:09.651369 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.651346    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 00:37:09.651568 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.651438    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2n49f\""
Jul 29 00:37:09.709961 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.709923    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/638b20e8-9b2b-4216-a1d0-d60a186ae13c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-vb4w7\" (UID: \"638b20e8-9b2b-4216-a1d0-d60a186ae13c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"
Jul 29 00:37:09.710560 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.710526    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9xg47"]
Jul 29 00:37:09.730439 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.730358    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9xg47"]
Jul 29 00:37:09.730694 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.730520    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.733837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.733808    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 00:37:09.733981 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.733806    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 00:37:09.734038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.734026    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 00:37:09.734106 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.734090    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-krv7p\""
Jul 29 00:37:09.737449 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.737428    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 00:37:09.777451 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.777410    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:09.810589 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.810552    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0eeb7a58-b6f6-497e-b239-635b3e67983e-crio-socket\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.810780 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.810598    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/638b20e8-9b2b-4216-a1d0-d60a186ae13c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-vb4w7\" (UID: \"638b20e8-9b2b-4216-a1d0-d60a186ae13c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"
Jul 29 00:37:09.810780 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.810621    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eeb7a58-b6f6-497e-b239-635b3e67983e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.810780 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:09.810709    2586 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 00:37:09.810916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.810776    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eeb7a58-b6f6-497e-b239-635b3e67983e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.810916 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:09.810805    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/638b20e8-9b2b-4216-a1d0-d60a186ae13c-tls-certificates podName:638b20e8-9b2b-4216-a1d0-d60a186ae13c nodeName:}" failed. No retries permitted until 2026-07-29 00:37:10.31078758 +0000 UTC m=+60.619715399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/638b20e8-9b2b-4216-a1d0-d60a186ae13c-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-vb4w7" (UID: "638b20e8-9b2b-4216-a1d0-d60a186ae13c") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 00:37:09.810916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.810853    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqwq\" (UniqueName: \"kubernetes.io/projected/0eeb7a58-b6f6-497e-b239-635b3e67983e-kube-api-access-pbqwq\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.810916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.810873    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eeb7a58-b6f6-497e-b239-635b3e67983e-data-volume\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.911842 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.911805    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqwq\" (UniqueName: \"kubernetes.io/projected/0eeb7a58-b6f6-497e-b239-635b3e67983e-kube-api-access-pbqwq\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.911842 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.911843    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eeb7a58-b6f6-497e-b239-635b3e67983e-data-volume\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.912084 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.911916    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0eeb7a58-b6f6-497e-b239-635b3e67983e-crio-socket\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.912084 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.911950    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eeb7a58-b6f6-497e-b239-635b3e67983e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.912084 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.911969    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eeb7a58-b6f6-497e-b239-635b3e67983e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.912084 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.912026    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0eeb7a58-b6f6-497e-b239-635b3e67983e-crio-socket\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.912084 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:09.912042    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 00:37:09.912394 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:09.912054    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 00:37:09.912394 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:09.912140    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0eeb7a58-b6f6-497e-b239-635b3e67983e-kube-rbac-proxy-cm podName:0eeb7a58-b6f6-497e-b239-635b3e67983e nodeName:}" failed. No retries permitted until 2026-07-29 00:37:10.412118428 +0000 UTC m=+60.721046254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0eeb7a58-b6f6-497e-b239-635b3e67983e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xg47" (UID: "0eeb7a58-b6f6-497e-b239-635b3e67983e") : configmap "kube-rbac-proxy" not found
Jul 29 00:37:09.912394 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:09.912159    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eeb7a58-b6f6-497e-b239-635b3e67983e-insights-runtime-extractor-tls podName:0eeb7a58-b6f6-497e-b239-635b3e67983e nodeName:}" failed. No retries permitted until 2026-07-29 00:37:10.412147449 +0000 UTC m=+60.721075270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0eeb7a58-b6f6-497e-b239-635b3e67983e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xg47" (UID: "0eeb7a58-b6f6-497e-b239-635b3e67983e") : secret "insights-runtime-extractor-tls" not found
Jul 29 00:37:09.912394 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.912291    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eeb7a58-b6f6-497e-b239-635b3e67983e-data-volume\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.919990 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.919963    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqwq\" (UniqueName: \"kubernetes.io/projected/0eeb7a58-b6f6-497e-b239-635b3e67983e-kube-api-access-pbqwq\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:09.962444 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:09.962409    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:10.065906 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.065881    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:10.163420 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.163389    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:10.314537 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.314493    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/638b20e8-9b2b-4216-a1d0-d60a186ae13c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-vb4w7\" (UID: \"638b20e8-9b2b-4216-a1d0-d60a186ae13c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"
Jul 29 00:37:10.317021 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.316968    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/638b20e8-9b2b-4216-a1d0-d60a186ae13c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-vb4w7\" (UID: \"638b20e8-9b2b-4216-a1d0-d60a186ae13c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"
Jul 29 00:37:10.372326 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.372295    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-llvgq"
Jul 29 00:37:10.377386 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.377353    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:10.415133 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.415090    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eeb7a58-b6f6-497e-b239-635b3e67983e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:10.415133 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.415133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eeb7a58-b6f6-497e-b239-635b3e67983e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:10.415770 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.415745    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eeb7a58-b6f6-497e-b239-635b3e67983e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:10.417579 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.417556    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eeb7a58-b6f6-497e-b239-635b3e67983e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xg47\" (UID: \"0eeb7a58-b6f6-497e-b239-635b3e67983e\") " pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:10.447917 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.447880    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc88b5bd6-fkcng" event={"ID":"81d23d92-4d3b-4ed0-98d1-d22e588537fa","Type":"ContainerStarted","Data":"3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56"}
Jul 29 00:37:10.448232 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.448211    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:10.467034 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.466980    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cc88b5bd6-fkcng" podStartSLOduration=2.466961629 podStartE2EDuration="2.466961629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:10.465472135 +0000 UTC m=+60.774399976" watchObservedRunningTime="2026-07-29 00:37:10.466961629 +0000 UTC m=+60.775889475"
Jul 29 00:37:10.561869 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.561841    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2n49f\""
Jul 29 00:37:10.568707 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.568644    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"
Jul 29 00:37:10.612877 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.612835    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:10.642974 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.642902    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-krv7p\""
Jul 29 00:37:10.650629 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.650605    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9xg47"
Jul 29 00:37:10.712693 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.712653    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:10.721488 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.721462    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"]
Jul 29 00:37:10.724579 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:10.724555    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638b20e8_9b2b_4216_a1d0_d60a186ae13c.slice/crio-773092605d9bff7d33fe91c9d954e20a88bb0c6f7463761303d9c21d99fc6b09 WatchSource:0}: Error finding container 773092605d9bff7d33fe91c9d954e20a88bb0c6f7463761303d9c21d99fc6b09: Status 404 returned error can't find the container with id 773092605d9bff7d33fe91c9d954e20a88bb0c6f7463761303d9c21d99fc6b09
Jul 29 00:37:10.780235 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:10.780175    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9xg47"]
Jul 29 00:37:10.783497 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:10.783476    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eeb7a58_b6f6_497e_b239_635b3e67983e.slice/crio-41dfc8b0e4316381f24fb2c7c2de21f0fbcbd0d6f0d3e7615a9363f39ec99c16 WatchSource:0}: Error finding container 41dfc8b0e4316381f24fb2c7c2de21f0fbcbd0d6f0d3e7615a9363f39ec99c16: Status 404 returned error can't find the container with id 41dfc8b0e4316381f24fb2c7c2de21f0fbcbd0d6f0d3e7615a9363f39ec99c16
Jul 29 00:37:11.453333 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:11.453291    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7" event={"ID":"638b20e8-9b2b-4216-a1d0-d60a186ae13c","Type":"ContainerStarted","Data":"773092605d9bff7d33fe91c9d954e20a88bb0c6f7463761303d9c21d99fc6b09"}
Jul 29 00:37:11.455090 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:11.455014    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xg47" event={"ID":"0eeb7a58-b6f6-497e-b239-635b3e67983e","Type":"ContainerStarted","Data":"41dfc8b0e4316381f24fb2c7c2de21f0fbcbd0d6f0d3e7615a9363f39ec99c16"}
Jul 29 00:37:11.559547 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:11.559475    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:11.655799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:11.655765    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:12.203568 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.203542    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cc88b5bd6-fkcng_81d23d92-4d3b-4ed0-98d1-d22e588537fa/console/0.log"
Jul 29 00:37:12.403662 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.403570    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84c48b94b9-cbwlb_a781e516-69f2-4186-a7c1-3af8465aeb0d/console/0.log"
Jul 29 00:37:12.460483 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.460447    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7" event={"ID":"638b20e8-9b2b-4216-a1d0-d60a186ae13c","Type":"ContainerStarted","Data":"56f1f5c2cf6934b2e94da402fda7fc35ba5458131f353ac6c193758668c039a2"}
Jul 29 00:37:12.460941 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.460663    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"
Jul 29 00:37:12.466446 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.466425    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7"
Jul 29 00:37:12.479367 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.479177    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vb4w7" podStartSLOduration=2.104640231 podStartE2EDuration="3.479157356s" totalImagesPullingTime="1.374517125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:12.477329212 +0000 UTC m=+62.786257052" watchObservedRunningTime="2026-07-29 00:37:12.479157356 +0000 UTC m=+62.788085200"
Jul 29 00:37:12.524321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.524283    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:12.613465 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.613415    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:12.685500 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.685414    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:12.708595 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.708562    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"]
Jul 29 00:37:12.710835 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.710808    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.713775 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.713752    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 29 00:37:12.714837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.714659    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 29 00:37:12.714837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.714666    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qndrr\""
Jul 29 00:37:12.714837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.714733    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 00:37:12.714837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.714756    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 00:37:12.714837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.714833    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 00:37:12.720743 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.720697    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"]
Jul 29 00:37:12.737940 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.737904    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21b3d742-675b-43c9-b0b5-bf0e2730f013-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.738114 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.737951    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88nv\" (UniqueName: \"kubernetes.io/projected/21b3d742-675b-43c9-b0b5-bf0e2730f013-kube-api-access-l88nv\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.738114 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.738030    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21b3d742-675b-43c9-b0b5-bf0e2730f013-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.738114 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.738079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21b3d742-675b-43c9-b0b5-bf0e2730f013-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.753316 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.753281    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:12.839523 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.839487    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21b3d742-675b-43c9-b0b5-bf0e2730f013-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.839729 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.839548    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88nv\" (UniqueName: \"kubernetes.io/projected/21b3d742-675b-43c9-b0b5-bf0e2730f013-kube-api-access-l88nv\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.839729 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.839591    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21b3d742-675b-43c9-b0b5-bf0e2730f013-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.839729 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.839618    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21b3d742-675b-43c9-b0b5-bf0e2730f013-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.839900 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:12.839774    2586 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 29 00:37:12.839900 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:12.839855    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b3d742-675b-43c9-b0b5-bf0e2730f013-prometheus-operator-tls podName:21b3d742-675b-43c9-b0b5-bf0e2730f013 nodeName:}" failed. No retries permitted until 2026-07-29 00:37:13.339834656 +0000 UTC m=+63.648762493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/21b3d742-675b-43c9-b0b5-bf0e2730f013-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-lp4qj" (UID: "21b3d742-675b-43c9-b0b5-bf0e2730f013") : secret "prometheus-operator-tls" not found
Jul 29 00:37:12.840369 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.840342    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21b3d742-675b-43c9-b0b5-bf0e2730f013-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.842417 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.842395    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21b3d742-675b-43c9-b0b5-bf0e2730f013-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:12.851622 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:12.851591    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88nv\" (UniqueName: \"kubernetes.io/projected/21b3d742-675b-43c9-b0b5-bf0e2730f013-kube-api-access-l88nv\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:13.327962 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:13.327928    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:13.343498 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:13.343460    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21b3d742-675b-43c9-b0b5-bf0e2730f013-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:13.346357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:13.346324    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21b3d742-675b-43c9-b0b5-bf0e2730f013-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lp4qj\" (UID: \"21b3d742-675b-43c9-b0b5-bf0e2730f013\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:13.603019 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:13.602931    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqrrw_b5e0eb10-33c8-4f4a-9076-f3ccda11708c/dns/0.log"
Jul 29 00:37:13.621884 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:13.621856    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"
Jul 29 00:37:13.803339 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:13.803307    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqrrw_b5e0eb10-33c8-4f4a-9076-f3ccda11708c/kube-rbac-proxy/0.log"
Jul 29 00:37:14.202022 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:14.201992    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-krk2j_a115e159-a003-4bdc-8e43-94ac470b5d1a/dns-node-resolver/0.log"
Jul 29 00:37:14.764878 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:14.764850    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:14.826394 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:14.826354    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:14.826620 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:14.826429    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:14.832276 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:14.832251    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:14.838529 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:14.838507    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:15.002552 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.002529    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-74znh_e7a2ae25-317e-403b-a257-b993fe994cf2/node-ca/0.log"
Jul 29 00:37:15.476256 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.476223    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:15.636748 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.636719    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:15.740545 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.740519    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lp4qj"]
Jul 29 00:37:15.743336 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:15.743311    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b3d742_675b_43c9_b0b5_bf0e2730f013.slice/crio-795d4b330944c22d9fc7e3d6dd76ab16441159b112eeaba84080bcfd57cb75f2 WatchSource:0}: Error finding container 795d4b330944c22d9fc7e3d6dd76ab16441159b112eeaba84080bcfd57cb75f2: Status 404 returned error can't find the container with id 795d4b330944c22d9fc7e3d6dd76ab16441159b112eeaba84080bcfd57cb75f2
Jul 29 00:37:15.803311 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.803280    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5cv9f_c12e1869-ccfe-438d-838c-e960413a9c1e/serve-healthcheck-canary/0.log"
Jul 29 00:37:15.855840 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.855812    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:15.956648 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.956620    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:15.967779 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.967752    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:37:15.967916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.967822    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:37:15.974415 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.974392    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 00:37:15.974716 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.974698    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 00:37:15.981046 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.981019    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5c14500d-ea5d-4433-ac3c-2df82c0ef982-original-pull-secret\") pod \"global-pull-secret-syncer-hrq9b\" (UID: \"5c14500d-ea5d-4433-ac3c-2df82c0ef982\") " pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:37:15.981046 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:15.981039    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c12d6fa-daae-4fc8-9325-1ae4a62765e9-metrics-certs\") pod \"network-metrics-daemon-2ljbv\" (UID: \"3c12d6fa-daae-4fc8-9325-1ae4a62765e9\") " pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:37:16.033392 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.033323    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4b5n\""
Jul 29 00:37:16.038669 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.038649    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrq9b"
Jul 29 00:37:16.040778 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.040755    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ljbv"
Jul 29 00:37:16.069355 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.069324    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:37:16.072406 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.072381    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 00:37:16.082603 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.082581    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 00:37:16.093509 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.093478    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj2s\" (UniqueName: \"kubernetes.io/projected/4b367dde-86b2-4108-9650-90d1d9b8e47a-kube-api-access-blj2s\") pod \"network-check-target-m2vg8\" (UID: \"4b367dde-86b2-4108-9650-90d1d9b8e47a\") " pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:37:16.211129 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.211089    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2ljbv"]
Jul 29 00:37:16.214608 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:16.214583    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c12d6fa_daae_4fc8_9325_1ae4a62765e9.slice/crio-303b87788519ff1cba017fa4b8432a21e875d08deead7082c623ce1bba210da2 WatchSource:0}: Error finding container 303b87788519ff1cba017fa4b8432a21e875d08deead7082c623ce1bba210da2: Status 404 returned error can't find the container with id 303b87788519ff1cba017fa4b8432a21e875d08deead7082c623ce1bba210da2
Jul 29 00:37:16.226238 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.226208    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrq9b"]
Jul 29 00:37:16.229711 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:16.229684    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c14500d_ea5d_4433_ac3c_2df82c0ef982.slice/crio-05a6e3270eb1c85f3a096f9f7272d06389b0d38397b83a5c54c389bf7748114f WatchSource:0}: Error finding container 05a6e3270eb1c85f3a096f9f7272d06389b0d38397b83a5c54c389bf7748114f: Status 404 returned error can't find the container with id 05a6e3270eb1c85f3a096f9f7272d06389b0d38397b83a5c54c389bf7748114f
Jul 29 00:37:16.343148 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.343105    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:16.349825 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.349793    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-44db9\""
Jul 29 00:37:16.358703 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.358448    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:37:16.480619 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.480583    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj" event={"ID":"21b3d742-675b-43c9-b0b5-bf0e2730f013","Type":"ContainerStarted","Data":"795d4b330944c22d9fc7e3d6dd76ab16441159b112eeaba84080bcfd57cb75f2"}
Jul 29 00:37:16.485167 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.484740    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xg47" event={"ID":"0eeb7a58-b6f6-497e-b239-635b3e67983e","Type":"ContainerStarted","Data":"1a08f24a8caf5d7f24b36e530b2689c5d17050abccc4fc71b072b1b3b9b7c7ba"}
Jul 29 00:37:16.487431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.487353    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ljbv" event={"ID":"3c12d6fa-daae-4fc8-9325-1ae4a62765e9","Type":"ContainerStarted","Data":"303b87788519ff1cba017fa4b8432a21e875d08deead7082c623ce1bba210da2"}
Jul 29 00:37:16.490312 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.490255    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrq9b" event={"ID":"5c14500d-ea5d-4433-ac3c-2df82c0ef982","Type":"ContainerStarted","Data":"05a6e3270eb1c85f3a096f9f7272d06389b0d38397b83a5c54c389bf7748114f"}
Jul 29 00:37:16.506572 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.506536    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:16.538998 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:16.538959    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2vg8"]
Jul 29 00:37:17.494894 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:17.494850    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2vg8" event={"ID":"4b367dde-86b2-4108-9650-90d1d9b8e47a","Type":"ContainerStarted","Data":"6361df972af083384dd205b23e4c5d78fe590f05600224ee551ac42bff902525"}
Jul 29 00:37:18.348576 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:18.348539    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:18.447711 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:18.447614    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:19.269120 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:19.268261    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:19.269120 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:19.268543    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:19.274928 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:19.274807    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:19.508170 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:19.508138    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:37:19.556472 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:19.556388    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c48b94b9-cbwlb"]
Jul 29 00:37:19.716546 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:19.716514    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:19.816671 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:19.816643    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:19.916914 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:19.916880    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:20.457526 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:20.457494    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sqrrw"
Jul 29 00:37:21.597361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:21.597329    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:21.699144 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:21.699057    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:21.839170 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:21.839131    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:21.939459 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:21.939423    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:22.833529 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:22.833475    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:23.053490 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.053407    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:23.155301 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.155262    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:23.521705 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.521670    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj" event={"ID":"21b3d742-675b-43c9-b0b5-bf0e2730f013","Type":"ContainerStarted","Data":"c655dce47c4a3a6a8f2e7524fc4b08ff7fe5e540f35ab4ad81fd67093662986b"}
Jul 29 00:37:23.521860 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.521778    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj" event={"ID":"21b3d742-675b-43c9-b0b5-bf0e2730f013","Type":"ContainerStarted","Data":"0a9ce929bf0e3e1d05be04247e7523cfe595798aadcda4eed8cd912a0c99e27a"}
Jul 29 00:37:23.523419 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.523394    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2vg8" event={"ID":"4b367dde-86b2-4108-9650-90d1d9b8e47a","Type":"ContainerStarted","Data":"fc80eca62b1b5e2e58fdb6be5a86ee2af193acaa6c4451a68ca2ea840df5b0ee"}
Jul 29 00:37:23.523555 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.523495    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:37:23.525417 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.525383    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xg47" event={"ID":"0eeb7a58-b6f6-497e-b239-635b3e67983e","Type":"ContainerStarted","Data":"9829f28dc958b33e7b0d774c27a6ed07181a6b225ca125c03bc8c88e8afb5527"}
Jul 29 00:37:23.527302 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.527273    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ljbv" event={"ID":"3c12d6fa-daae-4fc8-9325-1ae4a62765e9","Type":"ContainerStarted","Data":"42dee423fc22ebe739d3960f4d9674b9cab8fc091622ac273a9a273a1c72af69"}
Jul 29 00:37:23.527406 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.527332    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ljbv" event={"ID":"3c12d6fa-daae-4fc8-9325-1ae4a62765e9","Type":"ContainerStarted","Data":"57fb4c68dc37387bf492928ac954a1325349ff5a7e4bf09125e64bead894af17"}
Jul 29 00:37:23.528870 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.528842    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrq9b" event={"ID":"5c14500d-ea5d-4433-ac3c-2df82c0ef982","Type":"ContainerStarted","Data":"b6b86eff230c935ad76776a665886e2dbeaff662e94d6830228467dfeff74302"}
Jul 29 00:37:23.576761 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.576683    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2ljbv" podStartSLOduration=68.289399001 podStartE2EDuration="1m13.576659865s" totalImagesPullingTime="5.287260864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:23.576557939 +0000 UTC m=+73.885485782" watchObservedRunningTime="2026-07-29 00:37:23.576659865 +0000 UTC m=+73.885587709"
Jul 29 00:37:23.577741 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.577695    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-lp4qj" podStartSLOduration=5.901646882 podStartE2EDuration="11.577680057s" totalImagesPullingTime="5.676033175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:23.547038703 +0000 UTC m=+73.855966545" watchObservedRunningTime="2026-07-29 00:37:23.577680057 +0000 UTC m=+73.886607900"
Jul 29 00:37:23.598824 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.598769    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m2vg8" podStartSLOduration=67.633327497 podStartE2EDuration="1m13.598755919s" totalImagesPullingTime="5.965428422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:23.59746319 +0000 UTC m=+73.906391036" watchObservedRunningTime="2026-07-29 00:37:23.598755919 +0000 UTC m=+73.907683761"
Jul 29 00:37:23.617504 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.617438    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hrq9b" podStartSLOduration=67.401001177 podStartE2EDuration="1m13.617412307s" totalImagesPullingTime="6.21641113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:23.617124732 +0000 UTC m=+73.926052616" watchObservedRunningTime="2026-07-29 00:37:23.617412307 +0000 UTC m=+73.926340151"
Jul 29 00:37:23.683473 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.683439    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:23.785113 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:23.784996    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:24.837777 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:24.837741    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.074756 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.074716    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.139001 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.138930    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tgqz8"]
Jul 29 00:37:25.153348 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.153322    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.156290 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.156263    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nzh86\""
Jul 29 00:37:25.156431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.156340    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 00:37:25.156431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.156279    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 00:37:25.156431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.156263    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 00:37:25.156967 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.156943    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.231428 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.231400    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.246549 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.246505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-wtmp\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.246714 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.246564    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c542210c-ab65-4f1d-8573-37fe1ba9b955-sys\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.246714 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.246596    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-textfile\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.246714 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.246631    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c542210c-ab65-4f1d-8573-37fe1ba9b955-root\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.246714 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.246685    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.246949 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.246731    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-tls\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.246949 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.246770    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r4nw\" (UniqueName: \"kubernetes.io/projected/c542210c-ab65-4f1d-8573-37fe1ba9b955-kube-api-access-7r4nw\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.246949 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.246821    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.246949 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.246858    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c542210c-ab65-4f1d-8573-37fe1ba9b955-metrics-client-ca\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.347604 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347561    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c542210c-ab65-4f1d-8573-37fe1ba9b955-sys\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.347799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347618    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-textfile\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.347799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347664    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c542210c-ab65-4f1d-8573-37fe1ba9b955-root\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.347799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347693    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c542210c-ab65-4f1d-8573-37fe1ba9b955-sys\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.347799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347714    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.347799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347781    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-tls\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.348070 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347808    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r4nw\" (UniqueName: \"kubernetes.io/projected/c542210c-ab65-4f1d-8573-37fe1ba9b955-kube-api-access-7r4nw\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.348070 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347837    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.348070 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347891    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c542210c-ab65-4f1d-8573-37fe1ba9b955-metrics-client-ca\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.348070 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.347941    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-wtmp\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.348070 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.348027    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-textfile\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.348342 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.348096    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-wtmp\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.348342 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:25.348220    2586 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 00:37:25.348342 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.348292    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c542210c-ab65-4f1d-8573-37fe1ba9b955-root\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.348342 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:25.348331    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-tls podName:c542210c-ab65-4f1d-8573-37fe1ba9b955 nodeName:}" failed. No retries permitted until 2026-07-29 00:37:25.848308001 +0000 UTC m=+76.157235827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-tls") pod "node-exporter-tgqz8" (UID: "c542210c-ab65-4f1d-8573-37fe1ba9b955") : secret "node-exporter-tls" not found
Jul 29 00:37:25.348638 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.348610    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c542210c-ab65-4f1d-8573-37fe1ba9b955-metrics-client-ca\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.348781 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.348758    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.350557 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.350532    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.351538 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.351513    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.364603 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.364577    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r4nw\" (UniqueName: \"kubernetes.io/projected/c542210c-ab65-4f1d-8573-37fe1ba9b955-kube-api-access-7r4nw\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.446866 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.446778    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.548131 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.548085    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.651188 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.651150    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.794164 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.794078    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.853041 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.853001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-tls\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.855707 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.855678    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c542210c-ab65-4f1d-8573-37fe1ba9b955-node-exporter-tls\") pod \"node-exporter-tgqz8\" (UID: \"c542210c-ab65-4f1d-8573-37fe1ba9b955\") " pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:25.898124 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.898092    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:25.995172 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:25.995137    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:26.066220 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.065885    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tgqz8"
Jul 29 00:37:26.100175 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.100142    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:26.118362 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:26.118331    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc542210c_ab65_4f1d_8573_37fe1ba9b955.slice/crio-0cdcd4116708d9e7f54512a31beb7d72ee2193ba93823418d1bef6482c71045b WatchSource:0}: Error finding container 0cdcd4116708d9e7f54512a31beb7d72ee2193ba93823418d1bef6482c71045b: Status 404 returned error can't find the container with id 0cdcd4116708d9e7f54512a31beb7d72ee2193ba93823418d1bef6482c71045b
Jul 29 00:37:26.200043 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.200013    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:26.221612 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.221584    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 00:37:26.226255 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.226232    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.229479 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.229453    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 00:37:26.229630 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.229455    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 00:37:26.229630 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.229461    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 00:37:26.229740 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.229686    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 00:37:26.230170 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.230152    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 00:37:26.230283 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.230157    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 00:37:26.230283 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.230219    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dqcd7\""
Jul 29 00:37:26.230704 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.230680    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 00:37:26.230793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.230712    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 00:37:26.230793 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.230747    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 00:37:26.239799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.239444    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 00:37:26.302960 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.302931    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:26.357721 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357644    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a879a72-fe00-4a52-b335-29b33ec030c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.357721 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357685    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-web-config\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.357721 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357712    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a879a72-fe00-4a52-b335-29b33ec030c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.357958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357747    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.357958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357815    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a879a72-fe00-4a52-b335-29b33ec030c3-config-out\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.357958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357836    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.357958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357857    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.357958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357897    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.357958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357941    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.358225 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.357972    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll2qv\" (UniqueName: \"kubernetes.io/projected/1a879a72-fe00-4a52-b335-29b33ec030c3-kube-api-access-ll2qv\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.358225 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.358001    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.358225 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.358024    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1a879a72-fe00-4a52-b335-29b33ec030c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.358225 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.358052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a879a72-fe00-4a52-b335-29b33ec030c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.458778 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.458745    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a879a72-fe00-4a52-b335-29b33ec030c3-config-out\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.458778 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.458781    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459021 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.458802    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459021 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:26.458925    2586 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 29 00:37:26.459021 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.458958    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459021 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:26.459001    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-main-tls podName:1a879a72-fe00-4a52-b335-29b33ec030c3 nodeName:}" failed. No retries permitted until 2026-07-29 00:37:26.958975766 +0000 UTC m=+77.267903608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1a879a72-fe00-4a52-b335-29b33ec030c3") : secret "alertmanager-main-tls" not found
Jul 29 00:37:26.459240 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459042    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459240 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll2qv\" (UniqueName: \"kubernetes.io/projected/1a879a72-fe00-4a52-b335-29b33ec030c3-kube-api-access-ll2qv\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459240 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459099    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459240 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459123    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1a879a72-fe00-4a52-b335-29b33ec030c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459240 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459154    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a879a72-fe00-4a52-b335-29b33ec030c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459240 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459187    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a879a72-fe00-4a52-b335-29b33ec030c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459535 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459284    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-web-config\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459535 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459342    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a879a72-fe00-4a52-b335-29b33ec030c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459535 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459371    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.459667 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.459574    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1a879a72-fe00-4a52-b335-29b33ec030c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.460563 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.460491    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a879a72-fe00-4a52-b335-29b33ec030c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.460966 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.460913    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a879a72-fe00-4a52-b335-29b33ec030c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.462174 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.462150    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.462293 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.462272    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.462362 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.462277    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.462620 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.462601    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-web-config\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.462681 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.462616    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.462986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.462970    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a879a72-fe00-4a52-b335-29b33ec030c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.463274 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.463256    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a879a72-fe00-4a52-b335-29b33ec030c3-config-out\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.463327 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.463259    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.475807 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.475786    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll2qv\" (UniqueName: \"kubernetes.io/projected/1a879a72-fe00-4a52-b335-29b33ec030c3-kube-api-access-ll2qv\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.542678 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.542644    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xg47" event={"ID":"0eeb7a58-b6f6-497e-b239-635b3e67983e","Type":"ContainerStarted","Data":"365897db5dfda9a50c335f252856dc38cd210c0c454226e0c9a15a6a70337764"}
Jul 29 00:37:26.543711 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.543692    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgqz8" event={"ID":"c542210c-ab65-4f1d-8573-37fe1ba9b955","Type":"ContainerStarted","Data":"0cdcd4116708d9e7f54512a31beb7d72ee2193ba93823418d1bef6482c71045b"}
Jul 29 00:37:26.583231 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.583176    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9xg47" podStartSLOduration=2.365907542 podStartE2EDuration="17.583160626s" totalImagesPullingTime="15.217253084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:37:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:26.580545945 +0000 UTC m=+76.889473788" watchObservedRunningTime="2026-07-29 00:37:26.583160626 +0000 UTC m=+76.892088471"
Jul 29 00:37:26.678371 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.678298    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:26.821688 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.821652    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:26.942489 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.942402    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:26.964529 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.964494    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:26.967073 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:26.967026    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a879a72-fe00-4a52-b335-29b33ec030c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1a879a72-fe00-4a52-b335-29b33ec030c3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:27.141630 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:27.141596    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 00:37:27.277953 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:27.277931    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 00:37:27.279361 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:27.279335    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a879a72_fe00_4a52_b335_29b33ec030c3.slice/crio-7f67cfd44eda42589cdc08d0d7722e17d087108a3316b280927503bffc7ad2af WatchSource:0}: Error finding container 7f67cfd44eda42589cdc08d0d7722e17d087108a3316b280927503bffc7ad2af: Status 404 returned error can't find the container with id 7f67cfd44eda42589cdc08d0d7722e17d087108a3316b280927503bffc7ad2af
Jul 29 00:37:27.548825 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:27.548745    2586 generic.go:356] "Generic (PLEG): container finished" podID="c542210c-ab65-4f1d-8573-37fe1ba9b955" containerID="ad2e2d0d9e9cf823004860c83a5ba9428e8621dda5ec7b6fd6eb78a281da8828" exitCode=0
Jul 29 00:37:27.548982 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:27.548837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgqz8" event={"ID":"c542210c-ab65-4f1d-8573-37fe1ba9b955","Type":"ContainerDied","Data":"ad2e2d0d9e9cf823004860c83a5ba9428e8621dda5ec7b6fd6eb78a281da8828"}
Jul 29 00:37:27.549980 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:27.549957    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a879a72-fe00-4a52-b335-29b33ec030c3","Type":"ContainerStarted","Data":"7f67cfd44eda42589cdc08d0d7722e17d087108a3316b280927503bffc7ad2af"}
Jul 29 00:37:28.089904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:28.089865    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:28.189874 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:28.189836    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:28.289313 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:28.289276    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:28.501336 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:28.501313    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:28.555923 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:28.555892    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgqz8" event={"ID":"c542210c-ab65-4f1d-8573-37fe1ba9b955","Type":"ContainerStarted","Data":"7a4b8a666717a6450fc9995ddd86aaa278d3eb9cd24f6775a1ed992c3ade61c3"}
Jul 29 00:37:28.556074 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:28.555952    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgqz8" event={"ID":"c542210c-ab65-4f1d-8573-37fe1ba9b955","Type":"ContainerStarted","Data":"3b0d52e129478965cca27bd6cba6cc8c53a61eb22620f48e7f82f3634067c78c"}
Jul 29 00:37:28.557377 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:28.557352    2586 generic.go:356] "Generic (PLEG): container finished" podID="1a879a72-fe00-4a52-b335-29b33ec030c3" containerID="ea51b09c3c8eeb04f84bdd6d995510670a50c8cc8365631dee4116de8372532e" exitCode=0
Jul 29 00:37:28.557483 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:28.557429    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a879a72-fe00-4a52-b335-29b33ec030c3","Type":"ContainerDied","Data":"ea51b09c3c8eeb04f84bdd6d995510670a50c8cc8365631dee4116de8372532e"}
Jul 29 00:37:28.574930 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:28.574883    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tgqz8" podStartSLOduration=2.619378556 podStartE2EDuration="3.574869138s" totalImagesPullingTime="891.403394ms" totalInitContainerRuntime="64.087188ms" isStatefulPod=true podCreationTimestamp="2026-07-29 00:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:28.574093792 +0000 UTC m=+78.883021635" watchObservedRunningTime="2026-07-29 00:37:28.574869138 +0000 UTC m=+78.883796980"
Jul 29 00:37:29.867736 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.867704    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:29.876948 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.876924    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"]
Jul 29 00:37:29.880409 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.880388    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bbd645779-d4vrp"]
Jul 29 00:37:29.880545 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.880527    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:29.883287 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.883263    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 00:37:29.883404 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.883324    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t59bl\""
Jul 29 00:37:29.883404 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.883376    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:29.890363 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.890336    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"]
Jul 29 00:37:29.891985 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.891961    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbd645779-d4vrp"]
Jul 29 00:37:29.937104 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.937069    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:29.991547 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.991509    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fhc\" (UniqueName: \"kubernetes.io/projected/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-kube-api-access-86fhc\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:29.991735 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.991570    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-config\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:29.991735 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.991672    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-serving-cert\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:29.991844 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.991730    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd0a7f79-f912-476a-8779-e1590181fc39-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-cfqqp\" (UID: \"bd0a7f79-f912-476a-8779-e1590181fc39\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:29.991844 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.991779    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88778\" (UniqueName: \"kubernetes.io/projected/bd0a7f79-f912-476a-8779-e1590181fc39-kube-api-access-88778\") pod \"monitoring-plugin-68b7fc9574-cfqqp\" (UID: \"bd0a7f79-f912-476a-8779-e1590181fc39\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:29.991844 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.991805    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-trusted-ca-bundle\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:29.991844 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.991833    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-service-ca\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:29.992026 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.991895    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-oauth-config\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:29.992026 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:29.991918    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-oauth-serving-cert\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.053133 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.053099    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:30.093268 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.093228    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd0a7f79-f912-476a-8779-e1590181fc39-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-cfqqp\" (UID: \"bd0a7f79-f912-476a-8779-e1590181fc39\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:30.093433 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.093294    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88778\" (UniqueName: \"kubernetes.io/projected/bd0a7f79-f912-476a-8779-e1590181fc39-kube-api-access-88778\") pod \"monitoring-plugin-68b7fc9574-cfqqp\" (UID: \"bd0a7f79-f912-476a-8779-e1590181fc39\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:30.093433 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.093321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-trusted-ca-bundle\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.093433 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:30.093331    2586 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 29 00:37:30.093433 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.093344    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-service-ca\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.093433 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:30.093407    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd0a7f79-f912-476a-8779-e1590181fc39-monitoring-plugin-cert podName:bd0a7f79-f912-476a-8779-e1590181fc39 nodeName:}" failed. No retries permitted until 2026-07-29 00:37:30.593384502 +0000 UTC m=+80.902312325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/bd0a7f79-f912-476a-8779-e1590181fc39-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-cfqqp" (UID: "bd0a7f79-f912-476a-8779-e1590181fc39") : secret "monitoring-plugin-cert" not found
Jul 29 00:37:30.093433 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.093434    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-oauth-config\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.093829 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.093471    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-oauth-serving-cert\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.093829 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.093506    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fhc\" (UniqueName: \"kubernetes.io/projected/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-kube-api-access-86fhc\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.093829 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.093556    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-config\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.093829 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.093596    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-serving-cert\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.094360 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.094313    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-trusted-ca-bundle\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.094917 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.094861    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-service-ca\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.095036 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.095017    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-oauth-serving-cert\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.095105 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.095079    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-config\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.096384 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.096354    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-serving-cert\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.096473 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.096454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-oauth-config\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.103869 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.103845    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88778\" (UniqueName: \"kubernetes.io/projected/bd0a7f79-f912-476a-8779-e1590181fc39-kube-api-access-88778\") pod \"monitoring-plugin-68b7fc9574-cfqqp\" (UID: \"bd0a7f79-f912-476a-8779-e1590181fc39\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:30.104137 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.104111    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fhc\" (UniqueName: \"kubernetes.io/projected/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-kube-api-access-86fhc\") pod \"console-bbd645779-d4vrp\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") " pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.158148 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.158066    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:30.204772 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.204751    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:30.346863 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.346803    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbd645779-d4vrp"]
Jul 29 00:37:30.349739 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:30.349710    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1652e20_7f0e_471d_8acd_0d2faf1a5e27.slice/crio-9c39bf56e1d6ff598d1946c82bd5c96b8a033c69deca0bb96cf721da3a6cac58 WatchSource:0}: Error finding container 9c39bf56e1d6ff598d1946c82bd5c96b8a033c69deca0bb96cf721da3a6cac58: Status 404 returned error can't find the container with id 9c39bf56e1d6ff598d1946c82bd5c96b8a033c69deca0bb96cf721da3a6cac58
Jul 29 00:37:30.479821 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.479744    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:30.566443 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.566403    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbd645779-d4vrp" event={"ID":"a1652e20-7f0e-471d-8acd-0d2faf1a5e27","Type":"ContainerStarted","Data":"12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6"}
Jul 29 00:37:30.566603 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.566461    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbd645779-d4vrp" event={"ID":"a1652e20-7f0e-471d-8acd-0d2faf1a5e27","Type":"ContainerStarted","Data":"9c39bf56e1d6ff598d1946c82bd5c96b8a033c69deca0bb96cf721da3a6cac58"}
Jul 29 00:37:30.569158 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.569132    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a879a72-fe00-4a52-b335-29b33ec030c3","Type":"ContainerStarted","Data":"3f2652a67270b34f88b069812e725f29240951b2ecc923cfe7e8d10af2c8d1a0"}
Jul 29 00:37:30.569275 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.569188    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a879a72-fe00-4a52-b335-29b33ec030c3","Type":"ContainerStarted","Data":"3d3b2953ad657bd36fe3d889d46e69799d70ab7c1390d458e4f102f4081484ff"}
Jul 29 00:37:30.569275 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.569241    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a879a72-fe00-4a52-b335-29b33ec030c3","Type":"ContainerStarted","Data":"2b3a1572a7cc2e3777c9cb641f9ef9c4b3d891ced7c02391cb0216cd48d0fe4d"}
Jul 29 00:37:30.569275 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.569266    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a879a72-fe00-4a52-b335-29b33ec030c3","Type":"ContainerStarted","Data":"32aefc0bd8b1bb8b2e91531c28154beaef8b4632958a41b8ace343f86351bb8d"}
Jul 29 00:37:30.569410 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.569290    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a879a72-fe00-4a52-b335-29b33ec030c3","Type":"ContainerStarted","Data":"c160fdbdbfdeda3a613be277251c9692d3fc6c8a1197dc98b13910a881e7803b"}
Jul 29 00:37:30.591504 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.591453    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bbd645779-d4vrp" podStartSLOduration=1.591434502 podStartE2EDuration="1.591434502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:30.590750955 +0000 UTC m=+80.899678797" watchObservedRunningTime="2026-07-29 00:37:30.591434502 +0000 UTC m=+80.900362344"
Jul 29 00:37:30.597466 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.597440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd0a7f79-f912-476a-8779-e1590181fc39-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-cfqqp\" (UID: \"bd0a7f79-f912-476a-8779-e1590181fc39\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:30.599795 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.599774    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd0a7f79-f912-476a-8779-e1590181fc39-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-cfqqp\" (UID: \"bd0a7f79-f912-476a-8779-e1590181fc39\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:30.794946 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.794857    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:30.917742 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:30.917712    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"]
Jul 29 00:37:31.117920 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:31.117864    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:31.249809 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:31.249774    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:31.465091 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:31.464938    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:31.577619 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:31.577583    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp" event={"ID":"bd0a7f79-f912-476a-8779-e1590181fc39","Type":"ContainerStarted","Data":"7e7f9bb03acb86e4d75d5ee0acef0548e867228c6d64a94039d09ac473b740e5"}
Jul 29 00:37:31.581931 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:31.581169    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a879a72-fe00-4a52-b335-29b33ec030c3","Type":"ContainerStarted","Data":"0551394ff27387742dbe8bccded37d4c5ac2951a5dcbdfb15bcc9eadbdf491b5"}
Jul 29 00:37:31.621773 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:31.621713    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.087084682 podStartE2EDuration="5.621696383s" totalImagesPullingTime="3.519744806s" totalInitContainerRuntime="14.866895ms" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:31.620981397 +0000 UTC m=+81.929909241" watchObservedRunningTime="2026-07-29 00:37:31.621696383 +0000 UTC m=+81.930624224"
Jul 29 00:37:31.764609 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:31.764409    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:31.899668 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:31.899638    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:33.589754 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:33.589722    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp" event={"ID":"bd0a7f79-f912-476a-8779-e1590181fc39","Type":"ContainerStarted","Data":"5d9542a7ac90688ad80b6aaf2aaac0a43f8d6e293be5aa32527f426812dc33e3"}
Jul 29 00:37:33.590236 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:33.589924    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:33.594640 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:33.594620    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp"
Jul 29 00:37:33.605366 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:33.605316    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cfqqp" podStartSLOduration=2.4214777769999998 podStartE2EDuration="4.605300136s" totalImagesPullingTime="2.183822359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:33.604513743 +0000 UTC m=+83.913441586" watchObservedRunningTime="2026-07-29 00:37:33.605300136 +0000 UTC m=+83.914227979"
Jul 29 00:37:34.574788 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.574739    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbd645779-d4vrp"]
Jul 29 00:37:34.608122 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.608087    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65c8df4f5c-ff9m5"]
Jul 29 00:37:34.611440 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.611422    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.625936 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.625902    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c8df4f5c-ff9m5"]
Jul 29 00:37:34.733992 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.733946    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-service-ca\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.734173 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.734026    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-oauth-config\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.734173 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.734059    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-trusted-ca-bundle\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.734173 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.734111    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-config\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.734173 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.734162    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-oauth-serving-cert\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.734416 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.734222    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxpw\" (UniqueName: \"kubernetes.io/projected/50ed2fa2-e063-4fda-9dea-77b1edc151db-kube-api-access-whxpw\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.734416 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.734252    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-serving-cert\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.835262 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.835171    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-serving-cert\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.835262 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.835235    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-service-ca\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.835483 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.835277    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-oauth-config\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.835483 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.835296    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-trusted-ca-bundle\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.835483 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.835316    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-config\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.835483 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.835342    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-oauth-serving-cert\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.835483 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.835382    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxpw\" (UniqueName: \"kubernetes.io/projected/50ed2fa2-e063-4fda-9dea-77b1edc151db-kube-api-access-whxpw\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.836111 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.836084    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-config\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.836238 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.836085    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-service-ca\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.836238 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.836159    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-oauth-serving-cert\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.836238 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.836227    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-trusted-ca-bundle\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.838090 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.838063    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-oauth-config\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.838223 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.838189    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-serving-cert\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.843664 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.843637    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxpw\" (UniqueName: \"kubernetes.io/projected/50ed2fa2-e063-4fda-9dea-77b1edc151db-kube-api-access-whxpw\") pod \"console-65c8df4f5c-ff9m5\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") " pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:34.922549 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:34.922511    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:35.046700 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:35.046523    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c8df4f5c-ff9m5"]
Jul 29 00:37:35.049292 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:37:35.049258    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ed2fa2_e063_4fda_9dea_77b1edc151db.slice/crio-b9d22415a7344db0a35ea758db68084fd1693971a3c1bbaafa213704d57f164b WatchSource:0}: Error finding container b9d22415a7344db0a35ea758db68084fd1693971a3c1bbaafa213704d57f164b: Status 404 returned error can't find the container with id b9d22415a7344db0a35ea758db68084fd1693971a3c1bbaafa213704d57f164b
Jul 29 00:37:35.600237 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:35.600182    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c8df4f5c-ff9m5" event={"ID":"50ed2fa2-e063-4fda-9dea-77b1edc151db","Type":"ContainerStarted","Data":"a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593"}
Jul 29 00:37:35.600387 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:35.600259    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c8df4f5c-ff9m5" event={"ID":"50ed2fa2-e063-4fda-9dea-77b1edc151db","Type":"ContainerStarted","Data":"b9d22415a7344db0a35ea758db68084fd1693971a3c1bbaafa213704d57f164b"}
Jul 29 00:37:35.620518 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:35.620467    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65c8df4f5c-ff9m5" podStartSLOduration=1.620451413 podStartE2EDuration="1.620451413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:37:35.618963996 +0000 UTC m=+85.927891838" watchObservedRunningTime="2026-07-29 00:37:35.620451413 +0000 UTC m=+85.929379308"
Jul 29 00:37:35.802493 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:35.802458    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:35.866324 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:35.866244    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:38.048046 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:38.048012    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:38.102637 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:38.102605    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:40.205761 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:40.205723    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:40.755783 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:40.755743    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:40.809213 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:40.809172    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:44.580026 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.579990    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84c48b94b9-cbwlb" podUID="a781e516-69f2-4186-a7c1-3af8465aeb0d" containerName="console" containerID="cri-o://4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252" gracePeriod=15
Jul 29 00:37:44.819897 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.819855    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:44.922884 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.922786    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:44.922884 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.922876    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:44.923242 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.923223    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-serving-cert\") pod \"a781e516-69f2-4186-a7c1-3af8465aeb0d\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") "
Jul 29 00:37:44.923290 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.923266    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-service-ca\") pod \"a781e516-69f2-4186-a7c1-3af8465aeb0d\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") "
Jul 29 00:37:44.923290 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.923286    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-oauth-serving-cert\") pod \"a781e516-69f2-4186-a7c1-3af8465aeb0d\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") "
Jul 29 00:37:44.923364 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.923311    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-oauth-config\") pod \"a781e516-69f2-4186-a7c1-3af8465aeb0d\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") "
Jul 29 00:37:44.923364 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.923343    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-config\") pod \"a781e516-69f2-4186-a7c1-3af8465aeb0d\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") "
Jul 29 00:37:44.923452 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.923405    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z7qf\" (UniqueName: \"kubernetes.io/projected/a781e516-69f2-4186-a7c1-3af8465aeb0d-kube-api-access-6z7qf\") pod \"a781e516-69f2-4186-a7c1-3af8465aeb0d\" (UID: \"a781e516-69f2-4186-a7c1-3af8465aeb0d\") "
Jul 29 00:37:44.923810 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.923783    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-service-ca" (OuterVolumeSpecName: "service-ca") pod "a781e516-69f2-4186-a7c1-3af8465aeb0d" (UID: "a781e516-69f2-4186-a7c1-3af8465aeb0d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:37:44.923934 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.923776    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-config" (OuterVolumeSpecName: "console-config") pod "a781e516-69f2-4186-a7c1-3af8465aeb0d" (UID: "a781e516-69f2-4186-a7c1-3af8465aeb0d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:37:44.923998 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.923970    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a781e516-69f2-4186-a7c1-3af8465aeb0d" (UID: "a781e516-69f2-4186-a7c1-3af8465aeb0d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:37:44.925872 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.925844    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a781e516-69f2-4186-a7c1-3af8465aeb0d" (UID: "a781e516-69f2-4186-a7c1-3af8465aeb0d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:37:44.925972 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.925952    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a781e516-69f2-4186-a7c1-3af8465aeb0d" (UID: "a781e516-69f2-4186-a7c1-3af8465aeb0d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:37:44.926038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.925989    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a781e516-69f2-4186-a7c1-3af8465aeb0d-kube-api-access-6z7qf" (OuterVolumeSpecName: "kube-api-access-6z7qf") pod "a781e516-69f2-4186-a7c1-3af8465aeb0d" (UID: "a781e516-69f2-4186-a7c1-3af8465aeb0d"). InnerVolumeSpecName "kube-api-access-6z7qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:37:44.927785 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:44.927757    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:45.024886 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.024803    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:37:45.024886 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.024839    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:37:45.024886 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.024855    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-oauth-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:37:45.024886 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.024867    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-oauth-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:37:45.024886 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.024880    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a781e516-69f2-4186-a7c1-3af8465aeb0d-console-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:37:45.024886 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.024893    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z7qf\" (UniqueName: \"kubernetes.io/projected/a781e516-69f2-4186-a7c1-3af8465aeb0d-kube-api-access-6z7qf\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:37:45.638972 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.638936    2586 generic.go:356] "Generic (PLEG): container finished" podID="a781e516-69f2-4186-a7c1-3af8465aeb0d" containerID="4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252" exitCode=0
Jul 29 00:37:45.639495 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.638992    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c48b94b9-cbwlb"
Jul 29 00:37:45.639495 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.639036    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c48b94b9-cbwlb" event={"ID":"a781e516-69f2-4186-a7c1-3af8465aeb0d","Type":"ContainerDied","Data":"4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252"}
Jul 29 00:37:45.639495 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.639072    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c48b94b9-cbwlb" event={"ID":"a781e516-69f2-4186-a7c1-3af8465aeb0d","Type":"ContainerDied","Data":"71613fd8c77331e21f173c3c7f8f414435f7349904dc9a14719c48581dd8f9ad"}
Jul 29 00:37:45.639495 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.639090    2586 scope.go:117] "RemoveContainer" containerID="4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252"
Jul 29 00:37:45.643558 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.643530    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:37:45.649324 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.649304    2586 scope.go:117] "RemoveContainer" containerID="4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252"
Jul 29 00:37:45.649607 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:37:45.649588    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252\": container with ID starting with 4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252 not found: ID does not exist" containerID="4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252"
Jul 29 00:37:45.649658 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.649615    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252"} err="failed to get container status \"4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252\": rpc error: code = NotFound desc = could not find container \"4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252\": container with ID starting with 4ac355b14ddf19f8048b63b7fef4e16200d1303f665f290c24c47875569fe252 not found: ID does not exist"
Jul 29 00:37:45.677764 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.677730    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c48b94b9-cbwlb"]
Jul 29 00:37:45.682463 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.682435    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84c48b94b9-cbwlb"]
Jul 29 00:37:45.692900 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.692873    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc88b5bd6-fkcng"]
Jul 29 00:37:45.801608 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.801579    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:45.857772 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.857743    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:45.915665 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:45.915584    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:46.221380 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:46.221301    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a781e516-69f2-4186-a7c1-3af8465aeb0d" path="/var/lib/kubelet/pods/a781e516-69f2-4186-a7c1-3af8465aeb0d/volumes"
Jul 29 00:37:53.051296 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:53.051259    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:53.152456 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:53.152418    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:54.535099 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:54.535062    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m2vg8"
Jul 29 00:37:59.595423 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.595372    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bbd645779-d4vrp" podUID="a1652e20-7f0e-471d-8acd-0d2faf1a5e27" containerName="console" containerID="cri-o://12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6" gracePeriod=15
Jul 29 00:37:59.631742 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.631709    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:59.733884 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.733835    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:59.791321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.791289    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:37:59.841725 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.841695    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:37:59.928407 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928321    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-oauth-config\") pod \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") "
Jul 29 00:37:59.928407 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928357    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-trusted-ca-bundle\") pod \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") "
Jul 29 00:37:59.928407 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928386    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86fhc\" (UniqueName: \"kubernetes.io/projected/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-kube-api-access-86fhc\") pod \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") "
Jul 29 00:37:59.928696 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928433    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-service-ca\") pod \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") "
Jul 29 00:37:59.928696 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928557    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-oauth-serving-cert\") pod \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") "
Jul 29 00:37:59.928696 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928619    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-serving-cert\") pod \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") "
Jul 29 00:37:59.928696 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928648    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-config\") pod \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\" (UID: \"a1652e20-7f0e-471d-8acd-0d2faf1a5e27\") "
Jul 29 00:37:59.928901 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928854    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-service-ca" (OuterVolumeSpecName: "service-ca") pod "a1652e20-7f0e-471d-8acd-0d2faf1a5e27" (UID: "a1652e20-7f0e-471d-8acd-0d2faf1a5e27"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:37:59.928901 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928874    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a1652e20-7f0e-471d-8acd-0d2faf1a5e27" (UID: "a1652e20-7f0e-471d-8acd-0d2faf1a5e27"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:37:59.929010 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.928978    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a1652e20-7f0e-471d-8acd-0d2faf1a5e27" (UID: "a1652e20-7f0e-471d-8acd-0d2faf1a5e27"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:37:59.929095 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.929067    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-config" (OuterVolumeSpecName: "console-config") pod "a1652e20-7f0e-471d-8acd-0d2faf1a5e27" (UID: "a1652e20-7f0e-471d-8acd-0d2faf1a5e27"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:37:59.930689 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.930666    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a1652e20-7f0e-471d-8acd-0d2faf1a5e27" (UID: "a1652e20-7f0e-471d-8acd-0d2faf1a5e27"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:37:59.930808 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.930685    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-kube-api-access-86fhc" (OuterVolumeSpecName: "kube-api-access-86fhc") pod "a1652e20-7f0e-471d-8acd-0d2faf1a5e27" (UID: "a1652e20-7f0e-471d-8acd-0d2faf1a5e27"). InnerVolumeSpecName "kube-api-access-86fhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:37:59.930852 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:37:59.930807    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a1652e20-7f0e-471d-8acd-0d2faf1a5e27" (UID: "a1652e20-7f0e-471d-8acd-0d2faf1a5e27"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:38:00.029862 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.029807    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:00.029862 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.029851    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:00.029862 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.029862    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-console-oauth-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:00.029862 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.029871    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-trusted-ca-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:00.029862 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.029879    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86fhc\" (UniqueName: \"kubernetes.io/projected/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-kube-api-access-86fhc\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:00.030151 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.029889    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:00.030151 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.029897    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1652e20-7f0e-471d-8acd-0d2faf1a5e27-oauth-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:00.693850 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.693815    2586 generic.go:356] "Generic (PLEG): container finished" podID="a1652e20-7f0e-471d-8acd-0d2faf1a5e27" containerID="12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6" exitCode=0
Jul 29 00:38:00.694375 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.693878    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbd645779-d4vrp"
Jul 29 00:38:00.694375 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.693898    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbd645779-d4vrp" event={"ID":"a1652e20-7f0e-471d-8acd-0d2faf1a5e27","Type":"ContainerDied","Data":"12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6"}
Jul 29 00:38:00.694375 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.693978    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbd645779-d4vrp" event={"ID":"a1652e20-7f0e-471d-8acd-0d2faf1a5e27","Type":"ContainerDied","Data":"9c39bf56e1d6ff598d1946c82bd5c96b8a033c69deca0bb96cf721da3a6cac58"}
Jul 29 00:38:00.694375 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.693998    2586 scope.go:117] "RemoveContainer" containerID="12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6"
Jul 29 00:38:00.702070 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.702052    2586 scope.go:117] "RemoveContainer" containerID="12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6"
Jul 29 00:38:00.702353 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:38:00.702330    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6\": container with ID starting with 12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6 not found: ID does not exist" containerID="12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6"
Jul 29 00:38:00.702449 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.702358    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6"} err="failed to get container status \"12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6\": rpc error: code = NotFound desc = could not find container \"12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6\": container with ID starting with 12129e7eeba4214ad3df6a5b238dd01df8a248bcac2b3421428dd24ef9e10da6 not found: ID does not exist"
Jul 29 00:38:00.713701 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.713674    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbd645779-d4vrp"]
Jul 29 00:38:00.715990 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:00.715969    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bbd645779-d4vrp"]
Jul 29 00:38:02.221061 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:02.221024    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1652e20-7f0e-471d-8acd-0d2faf1a5e27" path="/var/lib/kubelet/pods/a1652e20-7f0e-471d-8acd-0d2faf1a5e27/volumes"
Jul 29 00:38:10.712000 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:10.711956    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cc88b5bd6-fkcng" podUID="81d23d92-4d3b-4ed0-98d1-d22e588537fa" containerName="console" containerID="cri-o://3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56" gracePeriod=15
Jul 29 00:38:10.957611 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:10.957584    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:38:11.014331 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014230    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-oauth-config\") pod \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") "
Jul 29 00:38:11.014331 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014276    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-config\") pod \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") "
Jul 29 00:38:11.014331 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014303    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc68p\" (UniqueName: \"kubernetes.io/projected/81d23d92-4d3b-4ed0-98d1-d22e588537fa-kube-api-access-cc68p\") pod \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") "
Jul 29 00:38:11.014331 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014328    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-serving-cert\") pod \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") "
Jul 29 00:38:11.014652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014357    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-trusted-ca-bundle\") pod \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") "
Jul 29 00:38:11.014652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014544    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-service-ca\") pod \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") "
Jul 29 00:38:11.014652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014610    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-oauth-serving-cert\") pod \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\" (UID: \"81d23d92-4d3b-4ed0-98d1-d22e588537fa\") "
Jul 29 00:38:11.014861 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014816    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "81d23d92-4d3b-4ed0-98d1-d22e588537fa" (UID: "81d23d92-4d3b-4ed0-98d1-d22e588537fa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:38:11.014861 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014825    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-config" (OuterVolumeSpecName: "console-config") pod "81d23d92-4d3b-4ed0-98d1-d22e588537fa" (UID: "81d23d92-4d3b-4ed0-98d1-d22e588537fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:38:11.015025 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.014882    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-service-ca" (OuterVolumeSpecName: "service-ca") pod "81d23d92-4d3b-4ed0-98d1-d22e588537fa" (UID: "81d23d92-4d3b-4ed0-98d1-d22e588537fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:38:11.015130 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.015095    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "81d23d92-4d3b-4ed0-98d1-d22e588537fa" (UID: "81d23d92-4d3b-4ed0-98d1-d22e588537fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:38:11.016734 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.016713    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d23d92-4d3b-4ed0-98d1-d22e588537fa-kube-api-access-cc68p" (OuterVolumeSpecName: "kube-api-access-cc68p") pod "81d23d92-4d3b-4ed0-98d1-d22e588537fa" (UID: "81d23d92-4d3b-4ed0-98d1-d22e588537fa"). InnerVolumeSpecName "kube-api-access-cc68p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:38:11.017041 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.017013    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "81d23d92-4d3b-4ed0-98d1-d22e588537fa" (UID: "81d23d92-4d3b-4ed0-98d1-d22e588537fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:38:11.017132 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.017056    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "81d23d92-4d3b-4ed0-98d1-d22e588537fa" (UID: "81d23d92-4d3b-4ed0-98d1-d22e588537fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:38:11.115344 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.115290    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-trusted-ca-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:11.115344 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.115338    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:11.115344 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.115352    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-oauth-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:11.115596 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.115365    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-oauth-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:11.115596 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.115377    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:11.115596 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.115389    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc68p\" (UniqueName: \"kubernetes.io/projected/81d23d92-4d3b-4ed0-98d1-d22e588537fa-kube-api-access-cc68p\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:11.115596 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.115401    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81d23d92-4d3b-4ed0-98d1-d22e588537fa-console-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:38:11.735295 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.735257    2586 generic.go:356] "Generic (PLEG): container finished" podID="81d23d92-4d3b-4ed0-98d1-d22e588537fa" containerID="3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56" exitCode=0
Jul 29 00:38:11.735295 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.735299    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc88b5bd6-fkcng"
Jul 29 00:38:11.735799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.735342    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc88b5bd6-fkcng" event={"ID":"81d23d92-4d3b-4ed0-98d1-d22e588537fa","Type":"ContainerDied","Data":"3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56"}
Jul 29 00:38:11.735799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.735401    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc88b5bd6-fkcng" event={"ID":"81d23d92-4d3b-4ed0-98d1-d22e588537fa","Type":"ContainerDied","Data":"7ee042b91fafad922d3c5b20ef1071293fa8ff86aa0f3a6b71dcacf52c4ec22a"}
Jul 29 00:38:11.735799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.735433    2586 scope.go:117] "RemoveContainer" containerID="3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56"
Jul 29 00:38:11.743471 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.743455    2586 scope.go:117] "RemoveContainer" containerID="3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56"
Jul 29 00:38:11.743763 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:38:11.743745    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56\": container with ID starting with 3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56 not found: ID does not exist" containerID="3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56"
Jul 29 00:38:11.743826 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.743780    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56"} err="failed to get container status \"3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56\": rpc error: code = NotFound desc = could not find container \"3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56\": container with ID starting with 3fa333fb7688f28ab1b0063a743c2c139ba0e7e6e07218dcc2628ec4aeee8e56 not found: ID does not exist"
Jul 29 00:38:11.755239 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.755208    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc88b5bd6-fkcng"]
Jul 29 00:38:11.758755 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:11.758732    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cc88b5bd6-fkcng"]
Jul 29 00:38:12.220620 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:12.220582    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d23d92-4d3b-4ed0-98d1-d22e588537fa" path="/var/lib/kubelet/pods/81d23d92-4d3b-4ed0-98d1-d22e588537fa/volumes"
Jul 29 00:38:22.960401 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:22.960360    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:39.870676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:39.870642    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:39.934353 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:39.934319    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:41.326238 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:41.326182    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:41.400601 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:41.400569    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:42.896951 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.896909    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d9fbffb86-gwb6j"]
Jul 29 00:38:42.897357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.897244    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1652e20-7f0e-471d-8acd-0d2faf1a5e27" containerName="console"
Jul 29 00:38:42.897357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.897259    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1652e20-7f0e-471d-8acd-0d2faf1a5e27" containerName="console"
Jul 29 00:38:42.897357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.897274    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a781e516-69f2-4186-a7c1-3af8465aeb0d" containerName="console"
Jul 29 00:38:42.897357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.897280    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a781e516-69f2-4186-a7c1-3af8465aeb0d" containerName="console"
Jul 29 00:38:42.897357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.897287    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d23d92-4d3b-4ed0-98d1-d22e588537fa" containerName="console"
Jul 29 00:38:42.897357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.897293    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d23d92-4d3b-4ed0-98d1-d22e588537fa" containerName="console"
Jul 29 00:38:42.897357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.897344    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a781e516-69f2-4186-a7c1-3af8465aeb0d" containerName="console"
Jul 29 00:38:42.897357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.897351    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1652e20-7f0e-471d-8acd-0d2faf1a5e27" containerName="console"
Jul 29 00:38:42.897357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.897358    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d23d92-4d3b-4ed0-98d1-d22e588537fa" containerName="console"
Jul 29 00:38:42.900687 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.900667    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:42.915721 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.915697    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9fbffb86-gwb6j"]
Jul 29 00:38:42.946369 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:42.946335    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:43.014036 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.014002    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:43.066356 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.066317    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-oauth-config\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.066491 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.066356    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfjxc\" (UniqueName: \"kubernetes.io/projected/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-kube-api-access-pfjxc\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.066491 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.066439    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-serving-cert\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.066491 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.066470    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-trusted-ca-bundle\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.066606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.066508    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-service-ca\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.066606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.066532    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-oauth-serving-cert\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.066606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.066573    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-config\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.168033 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.167933    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-oauth-config\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.168033 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.167984    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfjxc\" (UniqueName: \"kubernetes.io/projected/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-kube-api-access-pfjxc\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.168033 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.168023    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-serving-cert\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.168329 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.168048    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-trusted-ca-bundle\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.168329 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.168082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-service-ca\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.168329 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.168245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-oauth-serving-cert\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.168329 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.168293    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-config\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.169000 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.168975    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-trusted-ca-bundle\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.169000 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.168984    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-service-ca\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.169177 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.169011    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-config\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.169177 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.169060    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-oauth-serving-cert\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.170561 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.170538    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-serving-cert\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.170661 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.170638    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-oauth-config\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.177773 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.177745    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfjxc\" (UniqueName: \"kubernetes.io/projected/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-kube-api-access-pfjxc\") pod \"console-d9fbffb86-gwb6j\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") " pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.209945 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.209906    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:43.343012 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.342984    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9fbffb86-gwb6j"]
Jul 29 00:38:43.345470 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:38:43.345441    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5d7591_95dc_40c5_a9d0_da8acc5d8bda.slice/crio-d2ce6ee0f20119488b7ce03dff106160593a33cd24fe3231569cfd27ffdf1b59 WatchSource:0}: Error finding container d2ce6ee0f20119488b7ce03dff106160593a33cd24fe3231569cfd27ffdf1b59: Status 404 returned error can't find the container with id d2ce6ee0f20119488b7ce03dff106160593a33cd24fe3231569cfd27ffdf1b59
Jul 29 00:38:43.419248 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.419140    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:43.847971 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.847932    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9fbffb86-gwb6j" event={"ID":"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda","Type":"ContainerStarted","Data":"82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25"}
Jul 29 00:38:43.848184 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.847992    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9fbffb86-gwb6j" event={"ID":"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda","Type":"ContainerStarted","Data":"d2ce6ee0f20119488b7ce03dff106160593a33cd24fe3231569cfd27ffdf1b59"}
Jul 29 00:38:43.869688 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:43.869632    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d9fbffb86-gwb6j" podStartSLOduration=1.86961558 podStartE2EDuration="1.86961558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:38:43.867992486 +0000 UTC m=+154.176920330" watchObservedRunningTime="2026-07-29 00:38:43.86961558 +0000 UTC m=+154.178543422"
Jul 29 00:38:53.210608 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:53.210564    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:53.211079 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:53.210639    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:53.215489 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:53.215457    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:53.295255 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:53.295221    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:53.349797 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:53.349768    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:53.894553 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:53.894523    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:38:53.940423 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:53.940382    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c8df4f5c-ff9m5"]
Jul 29 00:38:54.060147 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:54.060119    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:54.161365 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:54.161288    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:38:54.223164 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:38:54.223135    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:39:06.833436 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:06.833347    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:39:08.070481 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:08.070446    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:39:08.171123 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:08.171064    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:39:18.960539 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:18.960473    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65c8df4f5c-ff9m5" podUID="50ed2fa2-e063-4fda-9dea-77b1edc151db" containerName="console" containerID="cri-o://a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593" gracePeriod=15
Jul 29 00:39:19.204142 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.204113    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:39:19.251512 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.251417    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whxpw\" (UniqueName: \"kubernetes.io/projected/50ed2fa2-e063-4fda-9dea-77b1edc151db-kube-api-access-whxpw\") pod \"50ed2fa2-e063-4fda-9dea-77b1edc151db\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") "
Jul 29 00:39:19.251512 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.251473    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-trusted-ca-bundle\") pod \"50ed2fa2-e063-4fda-9dea-77b1edc151db\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") "
Jul 29 00:39:19.251512 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.251508    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-oauth-config\") pod \"50ed2fa2-e063-4fda-9dea-77b1edc151db\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") "
Jul 29 00:39:19.251775 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.251525    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-oauth-serving-cert\") pod \"50ed2fa2-e063-4fda-9dea-77b1edc151db\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") "
Jul 29 00:39:19.251775 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.251548    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-service-ca\") pod \"50ed2fa2-e063-4fda-9dea-77b1edc151db\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") "
Jul 29 00:39:19.251775 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.251564    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-config\") pod \"50ed2fa2-e063-4fda-9dea-77b1edc151db\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") "
Jul 29 00:39:19.251775 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.251587    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-serving-cert\") pod \"50ed2fa2-e063-4fda-9dea-77b1edc151db\" (UID: \"50ed2fa2-e063-4fda-9dea-77b1edc151db\") "
Jul 29 00:39:19.252033 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.252001    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "50ed2fa2-e063-4fda-9dea-77b1edc151db" (UID: "50ed2fa2-e063-4fda-9dea-77b1edc151db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:39:19.252123 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.252023    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "50ed2fa2-e063-4fda-9dea-77b1edc151db" (UID: "50ed2fa2-e063-4fda-9dea-77b1edc151db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:39:19.252123 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.252035    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-service-ca" (OuterVolumeSpecName: "service-ca") pod "50ed2fa2-e063-4fda-9dea-77b1edc151db" (UID: "50ed2fa2-e063-4fda-9dea-77b1edc151db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:39:19.252123 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.252072    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-config" (OuterVolumeSpecName: "console-config") pod "50ed2fa2-e063-4fda-9dea-77b1edc151db" (UID: "50ed2fa2-e063-4fda-9dea-77b1edc151db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:39:19.253747 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.253714    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ed2fa2-e063-4fda-9dea-77b1edc151db-kube-api-access-whxpw" (OuterVolumeSpecName: "kube-api-access-whxpw") pod "50ed2fa2-e063-4fda-9dea-77b1edc151db" (UID: "50ed2fa2-e063-4fda-9dea-77b1edc151db"). InnerVolumeSpecName "kube-api-access-whxpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:39:19.253747 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.253726    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "50ed2fa2-e063-4fda-9dea-77b1edc151db" (UID: "50ed2fa2-e063-4fda-9dea-77b1edc151db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:39:19.253884 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.253795    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "50ed2fa2-e063-4fda-9dea-77b1edc151db" (UID: "50ed2fa2-e063-4fda-9dea-77b1edc151db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:39:19.352417 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.352362    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-trusted-ca-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:39:19.352417 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.352406    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-oauth-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:39:19.352417 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.352417    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-oauth-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:39:19.352417 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.352427    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:39:19.352417 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.352436    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:39:19.352713 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.352444    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50ed2fa2-e063-4fda-9dea-77b1edc151db-console-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:39:19.352713 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.352453    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxpw\" (UniqueName: \"kubernetes.io/projected/50ed2fa2-e063-4fda-9dea-77b1edc151db-kube-api-access-whxpw\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:39:19.984861 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.984826    2586 generic.go:356] "Generic (PLEG): container finished" podID="50ed2fa2-e063-4fda-9dea-77b1edc151db" containerID="a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593" exitCode=0
Jul 29 00:39:19.985379 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.984871    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c8df4f5c-ff9m5" event={"ID":"50ed2fa2-e063-4fda-9dea-77b1edc151db","Type":"ContainerDied","Data":"a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593"}
Jul 29 00:39:19.985379 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.984889    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c8df4f5c-ff9m5"
Jul 29 00:39:19.985379 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.984911    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c8df4f5c-ff9m5" event={"ID":"50ed2fa2-e063-4fda-9dea-77b1edc151db","Type":"ContainerDied","Data":"b9d22415a7344db0a35ea758db68084fd1693971a3c1bbaafa213704d57f164b"}
Jul 29 00:39:19.985379 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.984930    2586 scope.go:117] "RemoveContainer" containerID="a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593"
Jul 29 00:39:19.993978 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.993957    2586 scope.go:117] "RemoveContainer" containerID="a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593"
Jul 29 00:39:19.994288 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:39:19.994265    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593\": container with ID starting with a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593 not found: ID does not exist" containerID="a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593"
Jul 29 00:39:19.994341 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:19.994301    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593"} err="failed to get container status \"a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593\": rpc error: code = NotFound desc = could not find container \"a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593\": container with ID starting with a8df46fd6346cf46f342d8e1757e98a83d32aee606a71873ceb155eade2a2593 not found: ID does not exist"
Jul 29 00:39:20.008148 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:20.008116    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c8df4f5c-ff9m5"]
Jul 29 00:39:20.013140 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:20.013110    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65c8df4f5c-ff9m5"]
Jul 29 00:39:20.219923 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:20.219887    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50ed2fa2-e063-4fda-9dea-77b1edc151db" path="/var/lib/kubelet/pods/50ed2fa2-e063-4fda-9dea-77b1edc151db/volumes"
Jul 29 00:39:52.984584 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:52.984541    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:39:53.081387 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.081350    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:39:53.650257 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.650216    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"]
Jul 29 00:39:53.650496 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.650484    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ed2fa2-e063-4fda-9dea-77b1edc151db" containerName="console"
Jul 29 00:39:53.650539 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.650500    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ed2fa2-e063-4fda-9dea-77b1edc151db" containerName="console"
Jul 29 00:39:53.650571 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.650550    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ed2fa2-e063-4fda-9dea-77b1edc151db" containerName="console"
Jul 29 00:39:53.653520 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.653502    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.656274 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.656251    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9j5qs\""
Jul 29 00:39:53.656403 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.656251    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 00:39:53.656403 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.656251    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 00:39:53.663963 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.663939    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"]
Jul 29 00:39:53.812612 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.812552    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.812612 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.812620    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.812856 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.812678    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94tql\" (UniqueName: \"kubernetes.io/projected/fe6235a8-7281-4aff-ac32-0bc70023d63f-kube-api-access-94tql\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.913971 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.913864    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.913971 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.913925    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94tql\" (UniqueName: \"kubernetes.io/projected/fe6235a8-7281-4aff-ac32-0bc70023d63f-kube-api-access-94tql\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.913971 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.913964    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.914376 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.914355    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.914437 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.914357    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.923046 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.923021    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94tql\" (UniqueName: \"kubernetes.io/projected/fe6235a8-7281-4aff-ac32-0bc70023d63f-kube-api-access-94tql\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:53.963074 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:53.963038    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:39:54.094743 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:54.094706    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"]
Jul 29 00:39:54.097631 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:39:54.097603    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6235a8_7281_4aff_ac32_0bc70023d63f.slice/crio-3b29c774010ea1473c3f59ec244ea5654cb22673e1f7badaf5e88bf76f00344d WatchSource:0}: Error finding container 3b29c774010ea1473c3f59ec244ea5654cb22673e1f7badaf5e88bf76f00344d: Status 404 returned error can't find the container with id 3b29c774010ea1473c3f59ec244ea5654cb22673e1f7badaf5e88bf76f00344d
Jul 29 00:39:54.104693 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:39:54.104665    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5" event={"ID":"fe6235a8-7281-4aff-ac32-0bc70023d63f","Type":"ContainerStarted","Data":"3b29c774010ea1473c3f59ec244ea5654cb22673e1f7badaf5e88bf76f00344d"}
Jul 29 00:40:01.132757 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:01.132713    2586 generic.go:356] "Generic (PLEG): container finished" podID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerID="c5f49db58aeb843ff55067862e6efba7774786dca7d723c190790601e70c2ec1" exitCode=0
Jul 29 00:40:01.133181 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:01.132789    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5" event={"ID":"fe6235a8-7281-4aff-ac32-0bc70023d63f","Type":"ContainerDied","Data":"c5f49db58aeb843ff55067862e6efba7774786dca7d723c190790601e70c2ec1"}
Jul 29 00:40:04.146683 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:04.146647    2586 generic.go:356] "Generic (PLEG): container finished" podID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerID="9dcdb661a9e3fd1ac3e59c940f4695b97768d46a95b14ac32b02068bff5eebd9" exitCode=0
Jul 29 00:40:04.146683 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:04.146689    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5" event={"ID":"fe6235a8-7281-4aff-ac32-0bc70023d63f","Type":"ContainerDied","Data":"9dcdb661a9e3fd1ac3e59c940f4695b97768d46a95b14ac32b02068bff5eebd9"}
Jul 29 00:40:12.179873 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:12.179839    2586 generic.go:356] "Generic (PLEG): container finished" podID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerID="950deb0bba91baf3367b703dfee3727786b734b403b8a3cd99ab3e97a3348334" exitCode=0
Jul 29 00:40:12.180277 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:12.179932    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5" event={"ID":"fe6235a8-7281-4aff-ac32-0bc70023d63f","Type":"ContainerDied","Data":"950deb0bba91baf3367b703dfee3727786b734b403b8a3cd99ab3e97a3348334"}
Jul 29 00:40:13.304715 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.304685    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:40:13.380498 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.380459    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94tql\" (UniqueName: \"kubernetes.io/projected/fe6235a8-7281-4aff-ac32-0bc70023d63f-kube-api-access-94tql\") pod \"fe6235a8-7281-4aff-ac32-0bc70023d63f\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") "
Jul 29 00:40:13.380652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.380536    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-util\") pod \"fe6235a8-7281-4aff-ac32-0bc70023d63f\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") "
Jul 29 00:40:13.380652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.380599    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-bundle\") pod \"fe6235a8-7281-4aff-ac32-0bc70023d63f\" (UID: \"fe6235a8-7281-4aff-ac32-0bc70023d63f\") "
Jul 29 00:40:13.381300 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.381269    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-bundle" (OuterVolumeSpecName: "bundle") pod "fe6235a8-7281-4aff-ac32-0bc70023d63f" (UID: "fe6235a8-7281-4aff-ac32-0bc70023d63f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:40:13.382720 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.382698    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6235a8-7281-4aff-ac32-0bc70023d63f-kube-api-access-94tql" (OuterVolumeSpecName: "kube-api-access-94tql") pod "fe6235a8-7281-4aff-ac32-0bc70023d63f" (UID: "fe6235a8-7281-4aff-ac32-0bc70023d63f"). InnerVolumeSpecName "kube-api-access-94tql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:40:13.386047 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.386016    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-util" (OuterVolumeSpecName: "util") pod "fe6235a8-7281-4aff-ac32-0bc70023d63f" (UID: "fe6235a8-7281-4aff-ac32-0bc70023d63f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:40:13.481837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.481729    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-util\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:40:13.481837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.481788    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe6235a8-7281-4aff-ac32-0bc70023d63f-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:40:13.481837 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:13.481807    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94tql\" (UniqueName: \"kubernetes.io/projected/fe6235a8-7281-4aff-ac32-0bc70023d63f-kube-api-access-94tql\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:40:14.189765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:14.189730    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5" event={"ID":"fe6235a8-7281-4aff-ac32-0bc70023d63f","Type":"ContainerDied","Data":"3b29c774010ea1473c3f59ec244ea5654cb22673e1f7badaf5e88bf76f00344d"}
Jul 29 00:40:14.189765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:14.189765    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbxm5"
Jul 29 00:40:14.189972 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:14.189803    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b29c774010ea1473c3f59ec244ea5654cb22673e1f7badaf5e88bf76f00344d"
Jul 29 00:40:19.699688 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:19.699642    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:20.503566 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:20.503525    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:21.660736 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.660696    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"]
Jul 29 00:40:21.661261 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.660953    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerName="util"
Jul 29 00:40:21.661261 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.660963    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerName="util"
Jul 29 00:40:21.661261 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.660976    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerName="pull"
Jul 29 00:40:21.661261 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.660981    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerName="pull"
Jul 29 00:40:21.661261 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.660995    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerName="extract"
Jul 29 00:40:21.661261 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.661001    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerName="extract"
Jul 29 00:40:21.661261 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.661045    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6235a8-7281-4aff-ac32-0bc70023d63f" containerName="extract"
Jul 29 00:40:21.664288 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.664270    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"
Jul 29 00:40:21.669700 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.669673    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 29 00:40:21.669700 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.669676    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 29 00:40:21.669871 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.669757    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4ts9t\""
Jul 29 00:40:21.687958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.687926    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"]
Jul 29 00:40:21.747579 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.747540    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb452b53-51ef-41a9-bdd8-7ad155a92236-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c96bf\" (UID: \"bb452b53-51ef-41a9-bdd8-7ad155a92236\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"
Jul 29 00:40:21.747765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.747593    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9pm\" (UniqueName: \"kubernetes.io/projected/bb452b53-51ef-41a9-bdd8-7ad155a92236-kube-api-access-9b9pm\") pod \"cert-manager-operator-controller-manager-55c444bf86-c96bf\" (UID: \"bb452b53-51ef-41a9-bdd8-7ad155a92236\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"
Jul 29 00:40:21.848589 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.848549    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb452b53-51ef-41a9-bdd8-7ad155a92236-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c96bf\" (UID: \"bb452b53-51ef-41a9-bdd8-7ad155a92236\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"
Jul 29 00:40:21.848741 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.848600    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9pm\" (UniqueName: \"kubernetes.io/projected/bb452b53-51ef-41a9-bdd8-7ad155a92236-kube-api-access-9b9pm\") pod \"cert-manager-operator-controller-manager-55c444bf86-c96bf\" (UID: \"bb452b53-51ef-41a9-bdd8-7ad155a92236\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"
Jul 29 00:40:21.848977 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.848956    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb452b53-51ef-41a9-bdd8-7ad155a92236-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c96bf\" (UID: \"bb452b53-51ef-41a9-bdd8-7ad155a92236\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"
Jul 29 00:40:21.857884 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.857858    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9pm\" (UniqueName: \"kubernetes.io/projected/bb452b53-51ef-41a9-bdd8-7ad155a92236-kube-api-access-9b9pm\") pod \"cert-manager-operator-controller-manager-55c444bf86-c96bf\" (UID: \"bb452b53-51ef-41a9-bdd8-7ad155a92236\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"
Jul 29 00:40:21.972836 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:21.972741    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"
Jul 29 00:40:22.107681 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:22.107639    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf"]
Jul 29 00:40:22.111178 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:40:22.111145    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb452b53_51ef_41a9_bdd8_7ad155a92236.slice/crio-edbf6c8df60043196500a34ec57858e44b0b2483925e4d89f2dc712b11dd617c WatchSource:0}: Error finding container edbf6c8df60043196500a34ec57858e44b0b2483925e4d89f2dc712b11dd617c: Status 404 returned error can't find the container with id edbf6c8df60043196500a34ec57858e44b0b2483925e4d89f2dc712b11dd617c
Jul 29 00:40:22.219873 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:22.219837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf" event={"ID":"bb452b53-51ef-41a9-bdd8-7ad155a92236","Type":"ContainerStarted","Data":"edbf6c8df60043196500a34ec57858e44b0b2483925e4d89f2dc712b11dd617c"}
Jul 29 00:40:23.311350 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:23.311317    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:23.412319 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:23.412290    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:24.968278 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:24.968243    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:25.067696 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:25.067666    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:25.230825 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:25.230740    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf" event={"ID":"bb452b53-51ef-41a9-bdd8-7ad155a92236","Type":"ContainerStarted","Data":"798cc61343ee340a93a19d91a22d2eba87cc3e3c667d61aa8244c66c3dd30d41"}
Jul 29 00:40:25.251986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:25.251931    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c96bf" podStartSLOduration=2.158694846 podStartE2EDuration="4.251912943s" totalImagesPullingTime="2.093218097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:40:25.251500767 +0000 UTC m=+255.560428634" watchObservedRunningTime="2026-07-29 00:40:25.251912943 +0000 UTC m=+255.560840785"
Jul 29 00:40:25.664698 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:25.664660    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:25.771273 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:25.771231    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:26.176682 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.176648    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"]
Jul 29 00:40:26.178874 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.178857    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.181570 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.181540    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9j5qs\""
Jul 29 00:40:26.181704 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.181590    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 00:40:26.182713 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.182686    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 00:40:26.188373 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.188351    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"]
Jul 29 00:40:26.284635 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.284597    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpwp\" (UniqueName: \"kubernetes.io/projected/a800dd5c-dff2-461f-9a80-7f6443858226-kube-api-access-4fpwp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.284635 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.284643    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.284857 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.284670    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.385538 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.385492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpwp\" (UniqueName: \"kubernetes.io/projected/a800dd5c-dff2-461f-9a80-7f6443858226-kube-api-access-4fpwp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.385538 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.385545    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.385800 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.385581    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.385956 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.385934    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.386021 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.385987    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.394058 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.394024    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpwp\" (UniqueName: \"kubernetes.io/projected/a800dd5c-dff2-461f-9a80-7f6443858226-kube-api-access-4fpwp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.489021 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.488929    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:26.618360 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.618330    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"]
Jul 29 00:40:26.620414 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:40:26.620389    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda800dd5c_dff2_461f_9a80_7f6443858226.slice/crio-5eba790cc19b287839c33b749c33a570200507ef5965cf308a879c6f53c003cd WatchSource:0}: Error finding container 5eba790cc19b287839c33b749c33a570200507ef5965cf308a879c6f53c003cd: Status 404 returned error can't find the container with id 5eba790cc19b287839c33b749c33a570200507ef5965cf308a879c6f53c003cd
Jul 29 00:40:26.974986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:26.974955    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:27.239380 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.239292    2586 generic.go:356] "Generic (PLEG): container finished" podID="a800dd5c-dff2-461f-9a80-7f6443858226" containerID="3f17590332d5b8fab8b1503eb289f613364e6e8b0ef66896d1cce9bb15ac6a49" exitCode=0
Jul 29 00:40:27.239380 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.239349    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48" event={"ID":"a800dd5c-dff2-461f-9a80-7f6443858226","Type":"ContainerDied","Data":"3f17590332d5b8fab8b1503eb289f613364e6e8b0ef66896d1cce9bb15ac6a49"}
Jul 29 00:40:27.239774 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.239383    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48" event={"ID":"a800dd5c-dff2-461f-9a80-7f6443858226","Type":"ContainerStarted","Data":"5eba790cc19b287839c33b749c33a570200507ef5965cf308a879c6f53c003cd"}
Jul 29 00:40:27.370210 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.370165    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:27.741217 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.741165    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"]
Jul 29 00:40:27.743153 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.743136    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"
Jul 29 00:40:27.745883 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.745857    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 00:40:27.746027 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.745885    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4sm8r\""
Jul 29 00:40:27.746027 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.745861    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 00:40:27.754982 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.754958    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"]
Jul 29 00:40:27.795733 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.795694    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/012e3892-4b03-4a40-9ec7-adc206d17f52-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2zpwq\" (UID: \"012e3892-4b03-4a40-9ec7-adc206d17f52\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"
Jul 29 00:40:27.795931 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.795762    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw8jl\" (UniqueName: \"kubernetes.io/projected/012e3892-4b03-4a40-9ec7-adc206d17f52-kube-api-access-dw8jl\") pod \"cert-manager-cainjector-559cdd8c87-2zpwq\" (UID: \"012e3892-4b03-4a40-9ec7-adc206d17f52\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"
Jul 29 00:40:27.897088 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.897041    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/012e3892-4b03-4a40-9ec7-adc206d17f52-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2zpwq\" (UID: \"012e3892-4b03-4a40-9ec7-adc206d17f52\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"
Jul 29 00:40:27.897307 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.897111    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw8jl\" (UniqueName: \"kubernetes.io/projected/012e3892-4b03-4a40-9ec7-adc206d17f52-kube-api-access-dw8jl\") pod \"cert-manager-cainjector-559cdd8c87-2zpwq\" (UID: \"012e3892-4b03-4a40-9ec7-adc206d17f52\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"
Jul 29 00:40:27.910786 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.910749    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/012e3892-4b03-4a40-9ec7-adc206d17f52-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2zpwq\" (UID: \"012e3892-4b03-4a40-9ec7-adc206d17f52\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"
Jul 29 00:40:27.910960 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:27.910855    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw8jl\" (UniqueName: \"kubernetes.io/projected/012e3892-4b03-4a40-9ec7-adc206d17f52-kube-api-access-dw8jl\") pod \"cert-manager-cainjector-559cdd8c87-2zpwq\" (UID: \"012e3892-4b03-4a40-9ec7-adc206d17f52\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"
Jul 29 00:40:28.060312 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:28.060279    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"
Jul 29 00:40:28.209952 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:28.209914    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq"]
Jul 29 00:40:28.212631 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:40:28.212600    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod012e3892_4b03_4a40_9ec7_adc206d17f52.slice/crio-53489e406275335c5294e896bab6aa01ff51425070bb988fd280c0b86d3fbe41 WatchSource:0}: Error finding container 53489e406275335c5294e896bab6aa01ff51425070bb988fd280c0b86d3fbe41: Status 404 returned error can't find the container with id 53489e406275335c5294e896bab6aa01ff51425070bb988fd280c0b86d3fbe41
Jul 29 00:40:28.244747 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:28.244704    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq" event={"ID":"012e3892-4b03-4a40-9ec7-adc206d17f52","Type":"ContainerStarted","Data":"53489e406275335c5294e896bab6aa01ff51425070bb988fd280c0b86d3fbe41"}
Jul 29 00:40:29.251919 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:29.251880    2586 generic.go:356] "Generic (PLEG): container finished" podID="a800dd5c-dff2-461f-9a80-7f6443858226" containerID="1d7f0c888a221e347a5551f3c3c72b528be67a7cf6b0dfb22c09bff61c3c9665" exitCode=0
Jul 29 00:40:29.252422 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:29.251958    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48" event={"ID":"a800dd5c-dff2-461f-9a80-7f6443858226","Type":"ContainerDied","Data":"1d7f0c888a221e347a5551f3c3c72b528be67a7cf6b0dfb22c09bff61c3c9665"}
Jul 29 00:40:30.258838 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:30.258801    2586 generic.go:356] "Generic (PLEG): container finished" podID="a800dd5c-dff2-461f-9a80-7f6443858226" containerID="5d240af934a7d2d51b2ad6eb96341e338f7ea7105f8fb118c646707baa887ab4" exitCode=0
Jul 29 00:40:30.259287 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:30.258860    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48" event={"ID":"a800dd5c-dff2-461f-9a80-7f6443858226","Type":"ContainerDied","Data":"5d240af934a7d2d51b2ad6eb96341e338f7ea7105f8fb118c646707baa887ab4"}
Jul 29 00:40:30.311967 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:30.311927    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:30.369891 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:30.369835    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:31.264555 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.264516    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq" event={"ID":"012e3892-4b03-4a40-9ec7-adc206d17f52","Type":"ContainerStarted","Data":"9ba840928c4314f4db358533af66807197f7d9c047332de17c33b73bb35f40eb"}
Jul 29 00:40:31.290288 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.290241    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zpwq" podStartSLOduration=1.461488948 podStartE2EDuration="4.290185936s" totalImagesPullingTime="2.828696988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:40:31.288586078 +0000 UTC m=+261.597513919" watchObservedRunningTime="2026-07-29 00:40:31.290185936 +0000 UTC m=+261.599113778"
Jul 29 00:40:31.401253 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.401222    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:31.530302 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.530253    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-util\") pod \"a800dd5c-dff2-461f-9a80-7f6443858226\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") "
Jul 29 00:40:31.530494 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.530352    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fpwp\" (UniqueName: \"kubernetes.io/projected/a800dd5c-dff2-461f-9a80-7f6443858226-kube-api-access-4fpwp\") pod \"a800dd5c-dff2-461f-9a80-7f6443858226\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") "
Jul 29 00:40:31.530494 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.530396    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-bundle\") pod \"a800dd5c-dff2-461f-9a80-7f6443858226\" (UID: \"a800dd5c-dff2-461f-9a80-7f6443858226\") "
Jul 29 00:40:31.530841 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.530807    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-bundle" (OuterVolumeSpecName: "bundle") pod "a800dd5c-dff2-461f-9a80-7f6443858226" (UID: "a800dd5c-dff2-461f-9a80-7f6443858226"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:40:31.532855 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.532805    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a800dd5c-dff2-461f-9a80-7f6443858226-kube-api-access-4fpwp" (OuterVolumeSpecName: "kube-api-access-4fpwp") pod "a800dd5c-dff2-461f-9a80-7f6443858226" (UID: "a800dd5c-dff2-461f-9a80-7f6443858226"). InnerVolumeSpecName "kube-api-access-4fpwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:40:31.536562 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.536493    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-util" (OuterVolumeSpecName: "util") pod "a800dd5c-dff2-461f-9a80-7f6443858226" (UID: "a800dd5c-dff2-461f-9a80-7f6443858226"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:40:31.630890 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.630803    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:40:31.630890 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.630833    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a800dd5c-dff2-461f-9a80-7f6443858226-util\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:40:31.630890 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:31.630842    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpwp\" (UniqueName: \"kubernetes.io/projected/a800dd5c-dff2-461f-9a80-7f6443858226-kube-api-access-4fpwp\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:40:32.271359 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.271324    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48"
Jul 29 00:40:32.271754 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.271358    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjxq48" event={"ID":"a800dd5c-dff2-461f-9a80-7f6443858226","Type":"ContainerDied","Data":"5eba790cc19b287839c33b749c33a570200507ef5965cf308a879c6f53c003cd"}
Jul 29 00:40:32.271754 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.271403    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eba790cc19b287839c33b749c33a570200507ef5965cf308a879c6f53c003cd"
Jul 29 00:40:32.953979 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.953944    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c5fvd"]
Jul 29 00:40:32.954292 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.954277    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a800dd5c-dff2-461f-9a80-7f6443858226" containerName="pull"
Jul 29 00:40:32.954342 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.954296    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a800dd5c-dff2-461f-9a80-7f6443858226" containerName="pull"
Jul 29 00:40:32.954342 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.954311    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a800dd5c-dff2-461f-9a80-7f6443858226" containerName="extract"
Jul 29 00:40:32.954342 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.954319    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a800dd5c-dff2-461f-9a80-7f6443858226" containerName="extract"
Jul 29 00:40:32.954342 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.954325    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a800dd5c-dff2-461f-9a80-7f6443858226" containerName="util"
Jul 29 00:40:32.954342 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.954331    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a800dd5c-dff2-461f-9a80-7f6443858226" containerName="util"
Jul 29 00:40:32.954492 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.954380    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a800dd5c-dff2-461f-9a80-7f6443858226" containerName="extract"
Jul 29 00:40:32.957418 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.957400    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:32.960021 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.959999    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-sp85f\""
Jul 29 00:40:32.965862 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:32.965835    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c5fvd"]
Jul 29 00:40:33.040900 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.040853    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd63e9ea-7e37-496a-a094-11a5525b17d5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c5fvd\" (UID: \"dd63e9ea-7e37-496a-a094-11a5525b17d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:33.041091 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.040916    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2z6d\" (UniqueName: \"kubernetes.io/projected/dd63e9ea-7e37-496a-a094-11a5525b17d5-kube-api-access-q2z6d\") pod \"cert-manager-webhook-6486ffd997-c5fvd\" (UID: \"dd63e9ea-7e37-496a-a094-11a5525b17d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:33.142324 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.142276    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd63e9ea-7e37-496a-a094-11a5525b17d5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c5fvd\" (UID: \"dd63e9ea-7e37-496a-a094-11a5525b17d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:33.142534 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.142335    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2z6d\" (UniqueName: \"kubernetes.io/projected/dd63e9ea-7e37-496a-a094-11a5525b17d5-kube-api-access-q2z6d\") pod \"cert-manager-webhook-6486ffd997-c5fvd\" (UID: \"dd63e9ea-7e37-496a-a094-11a5525b17d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:33.151600 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.151569    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd63e9ea-7e37-496a-a094-11a5525b17d5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c5fvd\" (UID: \"dd63e9ea-7e37-496a-a094-11a5525b17d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:33.151742 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.151646    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2z6d\" (UniqueName: \"kubernetes.io/projected/dd63e9ea-7e37-496a-a094-11a5525b17d5-kube-api-access-q2z6d\") pod \"cert-manager-webhook-6486ffd997-c5fvd\" (UID: \"dd63e9ea-7e37-496a-a094-11a5525b17d5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:33.267039 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.266941    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:33.397466 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.397435    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c5fvd"]
Jul 29 00:40:33.401576 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:40:33.401549    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd63e9ea_7e37_496a_a094_11a5525b17d5.slice/crio-c7f4cd55009dcf778ce189087008f28b5d0b56bdd93beea38a3d5950efdbbed7 WatchSource:0}: Error finding container c7f4cd55009dcf778ce189087008f28b5d0b56bdd93beea38a3d5950efdbbed7: Status 404 returned error can't find the container with id c7f4cd55009dcf778ce189087008f28b5d0b56bdd93beea38a3d5950efdbbed7
Jul 29 00:40:33.643739 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.643705    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:33.744130 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:33.744099    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:34.282282 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:34.282243    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd" event={"ID":"dd63e9ea-7e37-496a-a094-11a5525b17d5","Type":"ContainerStarted","Data":"2aa0dea3665f35a4b67db22eece7913e804fbbea2eaf4e4cb9316a454b59db27"}
Jul 29 00:40:34.282282 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:34.282291    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd" event={"ID":"dd63e9ea-7e37-496a-a094-11a5525b17d5","Type":"ContainerStarted","Data":"c7f4cd55009dcf778ce189087008f28b5d0b56bdd93beea38a3d5950efdbbed7"}
Jul 29 00:40:34.282493 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:34.282405    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:34.302555 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:34.302500    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd" podStartSLOduration=2.302483804 podStartE2EDuration="2.302483804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:40:34.300107693 +0000 UTC m=+264.609035536" watchObservedRunningTime="2026-07-29 00:40:34.302483804 +0000 UTC m=+264.611411645"
Jul 29 00:40:35.779336 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:35.779303    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:36.331616 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.331569    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-54cq8"]
Jul 29 00:40:36.347564 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.347528    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-54cq8"
Jul 29 00:40:36.348377 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.348349    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-54cq8"]
Jul 29 00:40:36.351312 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.351062    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-s8xfx\""
Jul 29 00:40:36.467107 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.467067    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2q5\" (UniqueName: \"kubernetes.io/projected/b61551f7-a19a-48f6-9522-3084109ae5e4-kube-api-access-4v2q5\") pod \"cert-manager-55b4d954d5-54cq8\" (UID: \"b61551f7-a19a-48f6-9522-3084109ae5e4\") " pod="cert-manager/cert-manager-55b4d954d5-54cq8"
Jul 29 00:40:36.467321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.467119    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b61551f7-a19a-48f6-9522-3084109ae5e4-bound-sa-token\") pod \"cert-manager-55b4d954d5-54cq8\" (UID: \"b61551f7-a19a-48f6-9522-3084109ae5e4\") " pod="cert-manager/cert-manager-55b4d954d5-54cq8"
Jul 29 00:40:36.567745 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.567712    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2q5\" (UniqueName: \"kubernetes.io/projected/b61551f7-a19a-48f6-9522-3084109ae5e4-kube-api-access-4v2q5\") pod \"cert-manager-55b4d954d5-54cq8\" (UID: \"b61551f7-a19a-48f6-9522-3084109ae5e4\") " pod="cert-manager/cert-manager-55b4d954d5-54cq8"
Jul 29 00:40:36.567895 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.567754    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b61551f7-a19a-48f6-9522-3084109ae5e4-bound-sa-token\") pod \"cert-manager-55b4d954d5-54cq8\" (UID: \"b61551f7-a19a-48f6-9522-3084109ae5e4\") " pod="cert-manager/cert-manager-55b4d954d5-54cq8"
Jul 29 00:40:36.576764 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.576732    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b61551f7-a19a-48f6-9522-3084109ae5e4-bound-sa-token\") pod \"cert-manager-55b4d954d5-54cq8\" (UID: \"b61551f7-a19a-48f6-9522-3084109ae5e4\") " pod="cert-manager/cert-manager-55b4d954d5-54cq8"
Jul 29 00:40:36.576888 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.576870    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2q5\" (UniqueName: \"kubernetes.io/projected/b61551f7-a19a-48f6-9522-3084109ae5e4-kube-api-access-4v2q5\") pod \"cert-manager-55b4d954d5-54cq8\" (UID: \"b61551f7-a19a-48f6-9522-3084109ae5e4\") " pod="cert-manager/cert-manager-55b4d954d5-54cq8"
Jul 29 00:40:36.660682 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.660587    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-54cq8"
Jul 29 00:40:36.728220 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.728078    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:36.797959 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.797933    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-54cq8"]
Jul 29 00:40:36.800555 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:40:36.800525    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb61551f7_a19a_48f6_9522_3084109ae5e4.slice/crio-a0c094d43f379a8126eeb08a4c6322984ddf04514eafed4332d1218aaaa99d38 WatchSource:0}: Error finding container a0c094d43f379a8126eeb08a4c6322984ddf04514eafed4332d1218aaaa99d38: Status 404 returned error can't find the container with id a0c094d43f379a8126eeb08a4c6322984ddf04514eafed4332d1218aaaa99d38
Jul 29 00:40:36.806647 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:36.806621    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:37.295811 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:37.295771    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-54cq8" event={"ID":"b61551f7-a19a-48f6-9522-3084109ae5e4","Type":"ContainerStarted","Data":"d95136cba88d35c3ae68530a46053f5a2f4657ba2fca87a3a37183b63f6028d7"}
Jul 29 00:40:37.295811 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:37.295818    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-54cq8" event={"ID":"b61551f7-a19a-48f6-9522-3084109ae5e4","Type":"ContainerStarted","Data":"a0c094d43f379a8126eeb08a4c6322984ddf04514eafed4332d1218aaaa99d38"}
Jul 29 00:40:37.316524 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:37.316467    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-54cq8" podStartSLOduration=1.31644875 podStartE2EDuration="1.31644875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:40:37.316106053 +0000 UTC m=+267.625033895" watchObservedRunningTime="2026-07-29 00:40:37.31644875 +0000 UTC m=+267.625376591"
Jul 29 00:40:40.288689 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:40.288654    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c5fvd"
Jul 29 00:40:43.198541 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:43.198485    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:43.298500 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:43.298466    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:45.431166 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:45.431135    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:46.600072 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.600035    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"]
Jul 29 00:40:46.602370 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.602352    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.605084 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.605060    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9j5qs\""
Jul 29 00:40:46.605220 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.605094    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 00:40:46.605220 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.605127    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 00:40:46.611856 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.611831    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"]
Jul 29 00:40:46.747885 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.747841    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.748066 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.747918    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xwk\" (UniqueName: \"kubernetes.io/projected/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-kube-api-access-48xwk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.748066 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.747944    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.848627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.848588    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.848799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.848651    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xwk\" (UniqueName: \"kubernetes.io/projected/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-kube-api-access-48xwk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.848799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.848679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.849031 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.849011    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.849069 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.849032    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.859613 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.859550    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xwk\" (UniqueName: \"kubernetes.io/projected/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-kube-api-access-48xwk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:46.912554 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:46.912518    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:47.055528 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:47.055485    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"]
Jul 29 00:40:47.063232 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:40:47.063178    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9ac18c_c8a7_4e92_82ca_3c4c1a8c9225.slice/crio-613fe565beab8ede0f7d91400762858bd40e7eaa820d2a0567075844fa51dc9d WatchSource:0}: Error finding container 613fe565beab8ede0f7d91400762858bd40e7eaa820d2a0567075844fa51dc9d: Status 404 returned error can't find the container with id 613fe565beab8ede0f7d91400762858bd40e7eaa820d2a0567075844fa51dc9d
Jul 29 00:40:47.341893 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:47.341855    2586 generic.go:356] "Generic (PLEG): container finished" podID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerID="5d77e9abeafd58c28e54143ea0b58acdc498b2b0d1af0b4dbe8de6d62cf08dc1" exitCode=0
Jul 29 00:40:47.342224 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:47.341914    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts" event={"ID":"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225","Type":"ContainerDied","Data":"5d77e9abeafd58c28e54143ea0b58acdc498b2b0d1af0b4dbe8de6d62cf08dc1"}
Jul 29 00:40:47.342224 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:47.341951    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts" event={"ID":"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225","Type":"ContainerStarted","Data":"613fe565beab8ede0f7d91400762858bd40e7eaa820d2a0567075844fa51dc9d"}
Jul 29 00:40:48.347935 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:48.347895    2586 generic.go:356] "Generic (PLEG): container finished" podID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerID="d490e775c71479b1b9f1d9d6bcdd10ef56da4453f236edc427f6a9e7375faba5" exitCode=0
Jul 29 00:40:48.348354 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:48.347963    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts" event={"ID":"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225","Type":"ContainerDied","Data":"d490e775c71479b1b9f1d9d6bcdd10ef56da4453f236edc427f6a9e7375faba5"}
Jul 29 00:40:49.355061 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:49.355024    2586 generic.go:356] "Generic (PLEG): container finished" podID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerID="72b30be76e44ecc6dc3138adc164aa0af435e04adda72dbb8cea22eca63a2061" exitCode=0
Jul 29 00:40:49.355491 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:49.355103    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts" event={"ID":"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225","Type":"ContainerDied","Data":"72b30be76e44ecc6dc3138adc164aa0af435e04adda72dbb8cea22eca63a2061"}
Jul 29 00:40:50.487833 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.487810    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:50.580962 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.580919    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-util\") pod \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") "
Jul 29 00:40:50.580962 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.580969    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-bundle\") pod \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") "
Jul 29 00:40:50.581167 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.581073    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48xwk\" (UniqueName: \"kubernetes.io/projected/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-kube-api-access-48xwk\") pod \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\" (UID: \"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225\") "
Jul 29 00:40:50.582012 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.581986    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-bundle" (OuterVolumeSpecName: "bundle") pod "3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" (UID: "3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:40:50.583110 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.583079    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-kube-api-access-48xwk" (OuterVolumeSpecName: "kube-api-access-48xwk") pod "3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" (UID: "3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225"). InnerVolumeSpecName "kube-api-access-48xwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:40:50.586993 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.586969    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-util" (OuterVolumeSpecName: "util") pod "3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" (UID: "3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:40:50.682565 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.682486    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48xwk\" (UniqueName: \"kubernetes.io/projected/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-kube-api-access-48xwk\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:40:50.682565 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.682513    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-util\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:40:50.682565 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:50.682523    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:40:51.364244 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:51.364218    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts"
Jul 29 00:40:51.364426 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:51.364219    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfts" event={"ID":"3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225","Type":"ContainerDied","Data":"613fe565beab8ede0f7d91400762858bd40e7eaa820d2a0567075844fa51dc9d"}
Jul 29 00:40:51.364426 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:51.364324    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="613fe565beab8ede0f7d91400762858bd40e7eaa820d2a0567075844fa51dc9d"
Jul 29 00:40:57.717582 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:57.717549    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:58.660533 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.660498    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"]
Jul 29 00:40:58.660792 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.660781    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerName="extract"
Jul 29 00:40:58.660832 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.660794    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerName="extract"
Jul 29 00:40:58.660832 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.660804    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerName="util"
Jul 29 00:40:58.660832 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.660810    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerName="util"
Jul 29 00:40:58.660832 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.660817    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerName="pull"
Jul 29 00:40:58.660832 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.660823    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerName="pull"
Jul 29 00:40:58.661010 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.660874    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9ac18c-c8a7-4e92-82ca-3c4c1a8c9225" containerName="extract"
Jul 29 00:40:58.666650 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.666623    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.669887 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.669853    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 00:40:58.670018 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.669865    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 00:40:58.670018 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.669966    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 00:40:58.670152 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.670137    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mbnrz\""
Jul 29 00:40:58.670297 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.670282    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 00:40:58.691606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.691567    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"]
Jul 29 00:40:58.749826 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.749786    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/603ac805-e47a-454e-a1bf-8d761fbcf4f9-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.749826 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.749831    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/603ac805-e47a-454e-a1bf-8d761fbcf4f9-manifests\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.750418 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.749850    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/603ac805-e47a-454e-a1bf-8d761fbcf4f9-tmp\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.750418 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.749961    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/603ac805-e47a-454e-a1bf-8d761fbcf4f9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.750418 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.750026    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djhp\" (UniqueName: \"kubernetes.io/projected/603ac805-e47a-454e-a1bf-8d761fbcf4f9-kube-api-access-6djhp\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.851358 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.851320    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djhp\" (UniqueName: \"kubernetes.io/projected/603ac805-e47a-454e-a1bf-8d761fbcf4f9-kube-api-access-6djhp\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.851571 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.851403    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/603ac805-e47a-454e-a1bf-8d761fbcf4f9-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.851571 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.851441    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/603ac805-e47a-454e-a1bf-8d761fbcf4f9-manifests\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.851571 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.851462    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/603ac805-e47a-454e-a1bf-8d761fbcf4f9-tmp\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.851757 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.851595    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/603ac805-e47a-454e-a1bf-8d761fbcf4f9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.851851 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.851825    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/603ac805-e47a-454e-a1bf-8d761fbcf4f9-manifests\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.853937 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.853912    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/603ac805-e47a-454e-a1bf-8d761fbcf4f9-tmp\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.854059 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.854042    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/603ac805-e47a-454e-a1bf-8d761fbcf4f9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.854256 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.854234    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/603ac805-e47a-454e-a1bf-8d761fbcf4f9-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.865090 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.865062    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djhp\" (UniqueName: \"kubernetes.io/projected/603ac805-e47a-454e-a1bf-8d761fbcf4f9-kube-api-access-6djhp\") pod \"opendatahub-operator-controller-manager-5cf6bc8b6f-88px6\" (UID: \"603ac805-e47a-454e-a1bf-8d761fbcf4f9\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:58.925608 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.925520    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:58.976848 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:58.976805    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:40:59.026895 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:59.026860    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:59.125114 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:59.125082    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:59.125623 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:59.125600    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"]
Jul 29 00:40:59.128163 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:40:59.128136    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603ac805_e47a_454e_a1bf_8d761fbcf4f9.slice/crio-21b61566b250dc63466057027e898f748b8006d793a2f865549479eba043d480 WatchSource:0}: Error finding container 21b61566b250dc63466057027e898f748b8006d793a2f865549479eba043d480: Status 404 returned error can't find the container with id 21b61566b250dc63466057027e898f748b8006d793a2f865549479eba043d480
Jul 29 00:40:59.199958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:59.199869    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:59.300650 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:59.300612    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:59.397442 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:59.397406    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6" event={"ID":"603ac805-e47a-454e-a1bf-8d761fbcf4f9","Type":"ContainerStarted","Data":"21b61566b250dc63466057027e898f748b8006d793a2f865549479eba043d480"}
Jul 29 00:40:59.400661 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:59.400639    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:40:59.505028 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:40:59.504944    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:02.414842 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:02.414810    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6" event={"ID":"603ac805-e47a-454e-a1bf-8d761fbcf4f9","Type":"ContainerStarted","Data":"64cdeaf2297ee96e114b26bb47f0802e101ade11f659ae42d4e2f30277b3f7fd"}
Jul 29 00:41:03.424061 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:03.424024    2586 generic.go:356] "Generic (PLEG): container finished" podID="603ac805-e47a-454e-a1bf-8d761fbcf4f9" containerID="64cdeaf2297ee96e114b26bb47f0802e101ade11f659ae42d4e2f30277b3f7fd" exitCode=0
Jul 29 00:41:03.424496 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:03.424111    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6" event={"ID":"603ac805-e47a-454e-a1bf-8d761fbcf4f9","Type":"ContainerDied","Data":"64cdeaf2297ee96e114b26bb47f0802e101ade11f659ae42d4e2f30277b3f7fd"}
Jul 29 00:41:03.711811 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:03.711733    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:03.843878 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:03.843842    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:04.430501 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:04.430466    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6" event={"ID":"603ac805-e47a-454e-a1bf-8d761fbcf4f9","Type":"ContainerStarted","Data":"e0739a57f69ba78894b0bf237bb2d97585d12cc36222602506d13caf3824f212"}
Jul 29 00:41:04.430939 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:04.430577    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:41:04.501910 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:04.501847    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6" podStartSLOduration=1.7326014889999999 podStartE2EDuration="6.501826151s" totalImagesPullingTime="3.75812298s" totalInitContainerRuntime="1.011101682s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:40:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:41:04.494649705 +0000 UTC m=+294.803577557" watchObservedRunningTime="2026-07-29 00:41:04.501826151 +0000 UTC m=+294.810754009"
Jul 29 00:41:09.908745 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:09.908711    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:10.009334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:10.009295    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:10.084357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:10.084312    2586 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 00:41:15.438660 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:15.438628    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cf6bc8b6f-88px6"
Jul 29 00:41:17.865261 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:17.865191    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:17.983413 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:17.983367    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:18.707601 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.707565    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jbxvv"]
Jul 29 00:41:18.712666 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.712637    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:18.715652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.715621    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 29 00:41:18.715652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.715640    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qhzjm\""
Jul 29 00:41:18.728312 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.728276    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jbxvv"]
Jul 29 00:41:18.748014 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.747985    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:18.810112 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.810073    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f2bd4e2-4712-4840-800f-eb424f728dc9-cert\") pod \"odh-model-controller-7998c67dc4-jbxvv\" (UID: \"5f2bd4e2-4712-4840-800f-eb424f728dc9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:18.810297 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.810153    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl2jk\" (UniqueName: \"kubernetes.io/projected/5f2bd4e2-4712-4840-800f-eb424f728dc9-kube-api-access-wl2jk\") pod \"odh-model-controller-7998c67dc4-jbxvv\" (UID: \"5f2bd4e2-4712-4840-800f-eb424f728dc9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:18.839568 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.839529    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"]
Jul 29 00:41:18.842894 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.842872    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:18.846329 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.846305    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 00:41:18.846588 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.846575    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 00:41:18.846631 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.846578    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 00:41:18.846844 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.846823    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 00:41:18.846962 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.846863    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kjwjd\""
Jul 29 00:41:18.861804 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.861775    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"]
Jul 29 00:41:18.882917 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.882879    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:18.910668 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.910630    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5373cb16-6de6-4fc9-a6b8-a7b7207dc945-tmp\") pod \"kube-auth-proxy-75d55f5564-tbqst\" (UID: \"5373cb16-6de6-4fc9-a6b8-a7b7207dc945\") " pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:18.910847 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.910691    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f2bd4e2-4712-4840-800f-eb424f728dc9-cert\") pod \"odh-model-controller-7998c67dc4-jbxvv\" (UID: \"5f2bd4e2-4712-4840-800f-eb424f728dc9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:18.910847 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.910735    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5373cb16-6de6-4fc9-a6b8-a7b7207dc945-tls-certs\") pod \"kube-auth-proxy-75d55f5564-tbqst\" (UID: \"5373cb16-6de6-4fc9-a6b8-a7b7207dc945\") " pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:18.910847 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.910757    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8lz\" (UniqueName: \"kubernetes.io/projected/5373cb16-6de6-4fc9-a6b8-a7b7207dc945-kube-api-access-4q8lz\") pod \"kube-auth-proxy-75d55f5564-tbqst\" (UID: \"5373cb16-6de6-4fc9-a6b8-a7b7207dc945\") " pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:18.910847 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.910830    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl2jk\" (UniqueName: \"kubernetes.io/projected/5f2bd4e2-4712-4840-800f-eb424f728dc9-kube-api-access-wl2jk\") pod \"odh-model-controller-7998c67dc4-jbxvv\" (UID: \"5f2bd4e2-4712-4840-800f-eb424f728dc9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:18.911040 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:41:18.910884    2586 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 29 00:41:18.911040 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:41:18.911004    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2bd4e2-4712-4840-800f-eb424f728dc9-cert podName:5f2bd4e2-4712-4840-800f-eb424f728dc9 nodeName:}" failed. No retries permitted until 2026-07-29 00:41:19.410981022 +0000 UTC m=+309.719908856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f2bd4e2-4712-4840-800f-eb424f728dc9-cert") pod "odh-model-controller-7998c67dc4-jbxvv" (UID: "5f2bd4e2-4712-4840-800f-eb424f728dc9") : secret "odh-model-controller-webhook-cert" not found
Jul 29 00:41:18.923453 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.923427    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl2jk\" (UniqueName: \"kubernetes.io/projected/5f2bd4e2-4712-4840-800f-eb424f728dc9-kube-api-access-wl2jk\") pod \"odh-model-controller-7998c67dc4-jbxvv\" (UID: \"5f2bd4e2-4712-4840-800f-eb424f728dc9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:18.970996 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:18.970907    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:19.012305 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.012269    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5373cb16-6de6-4fc9-a6b8-a7b7207dc945-tls-certs\") pod \"kube-auth-proxy-75d55f5564-tbqst\" (UID: \"5373cb16-6de6-4fc9-a6b8-a7b7207dc945\") " pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:19.012305 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.012305    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8lz\" (UniqueName: \"kubernetes.io/projected/5373cb16-6de6-4fc9-a6b8-a7b7207dc945-kube-api-access-4q8lz\") pod \"kube-auth-proxy-75d55f5564-tbqst\" (UID: \"5373cb16-6de6-4fc9-a6b8-a7b7207dc945\") " pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:19.012531 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.012362    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5373cb16-6de6-4fc9-a6b8-a7b7207dc945-tmp\") pod \"kube-auth-proxy-75d55f5564-tbqst\" (UID: \"5373cb16-6de6-4fc9-a6b8-a7b7207dc945\") " pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:19.014672 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.014647    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5373cb16-6de6-4fc9-a6b8-a7b7207dc945-tmp\") pod \"kube-auth-proxy-75d55f5564-tbqst\" (UID: \"5373cb16-6de6-4fc9-a6b8-a7b7207dc945\") " pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:19.014952 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.014932    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5373cb16-6de6-4fc9-a6b8-a7b7207dc945-tls-certs\") pod \"kube-auth-proxy-75d55f5564-tbqst\" (UID: \"5373cb16-6de6-4fc9-a6b8-a7b7207dc945\") " pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:19.021431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.021405    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8lz\" (UniqueName: \"kubernetes.io/projected/5373cb16-6de6-4fc9-a6b8-a7b7207dc945-kube-api-access-4q8lz\") pod \"kube-auth-proxy-75d55f5564-tbqst\" (UID: \"5373cb16-6de6-4fc9-a6b8-a7b7207dc945\") " pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:19.073477 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.073425    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:19.152737 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.152695    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"
Jul 29 00:41:19.180597 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.180554    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:19.305537 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.305507    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75d55f5564-tbqst"]
Jul 29 00:41:19.308732 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:41:19.308705    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5373cb16_6de6_4fc9_a6b8_a7b7207dc945.slice/crio-59dd60631f9623b7d79fc6fe37a2df95166864d09570b1eb93cc6afb7d49628f WatchSource:0}: Error finding container 59dd60631f9623b7d79fc6fe37a2df95166864d09570b1eb93cc6afb7d49628f: Status 404 returned error can't find the container with id 59dd60631f9623b7d79fc6fe37a2df95166864d09570b1eb93cc6afb7d49628f
Jul 29 00:41:19.310672 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.310652    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 00:41:19.415566 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.415517    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f2bd4e2-4712-4840-800f-eb424f728dc9-cert\") pod \"odh-model-controller-7998c67dc4-jbxvv\" (UID: \"5f2bd4e2-4712-4840-800f-eb424f728dc9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:19.418395 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.418365    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f2bd4e2-4712-4840-800f-eb424f728dc9-cert\") pod \"odh-model-controller-7998c67dc4-jbxvv\" (UID: \"5f2bd4e2-4712-4840-800f-eb424f728dc9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:19.502331 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.502229    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst" event={"ID":"5373cb16-6de6-4fc9-a6b8-a7b7207dc945","Type":"ContainerStarted","Data":"59dd60631f9623b7d79fc6fe37a2df95166864d09570b1eb93cc6afb7d49628f"}
Jul 29 00:41:19.526403 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.526360    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zrpn4"]
Jul 29 00:41:19.531055 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.531029    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:19.543605 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.543577    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 29 00:41:19.570115 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.570081    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zrpn4"]
Jul 29 00:41:19.574465 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.574426    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:19.617020 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.616985    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b02ca09-a952-49e2-8df8-40f0d7c716f6-tls-certs\") pod \"model-serving-api-54d89bd5fc-zrpn4\" (UID: \"9b02ca09-a952-49e2-8df8-40f0d7c716f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:19.617230 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.617079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2x82\" (UniqueName: \"kubernetes.io/projected/9b02ca09-a952-49e2-8df8-40f0d7c716f6-kube-api-access-f2x82\") pod \"model-serving-api-54d89bd5fc-zrpn4\" (UID: \"9b02ca09-a952-49e2-8df8-40f0d7c716f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:19.622921 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.622888    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:19.717731 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.717696    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2x82\" (UniqueName: \"kubernetes.io/projected/9b02ca09-a952-49e2-8df8-40f0d7c716f6-kube-api-access-f2x82\") pod \"model-serving-api-54d89bd5fc-zrpn4\" (UID: \"9b02ca09-a952-49e2-8df8-40f0d7c716f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:19.717914 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.717784    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b02ca09-a952-49e2-8df8-40f0d7c716f6-tls-certs\") pod \"model-serving-api-54d89bd5fc-zrpn4\" (UID: \"9b02ca09-a952-49e2-8df8-40f0d7c716f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:19.720460 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.720432    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b02ca09-a952-49e2-8df8-40f0d7c716f6-tls-certs\") pod \"model-serving-api-54d89bd5fc-zrpn4\" (UID: \"9b02ca09-a952-49e2-8df8-40f0d7c716f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:19.745086 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.745007    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2x82\" (UniqueName: \"kubernetes.io/projected/9b02ca09-a952-49e2-8df8-40f0d7c716f6-kube-api-access-f2x82\") pod \"model-serving-api-54d89bd5fc-zrpn4\" (UID: \"9b02ca09-a952-49e2-8df8-40f0d7c716f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:19.746173 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.746130    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:19.794320 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.794291    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jbxvv"]
Jul 29 00:41:19.795589 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:41:19.795560    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2bd4e2_4712_4840_800f_eb424f728dc9.slice/crio-5a501b28c027dbdac98499af049475c21ecf299e305c6c07be35cb45f8b4dfdb WatchSource:0}: Error finding container 5a501b28c027dbdac98499af049475c21ecf299e305c6c07be35cb45f8b4dfdb: Status 404 returned error can't find the container with id 5a501b28c027dbdac98499af049475c21ecf299e305c6c07be35cb45f8b4dfdb
Jul 29 00:41:19.841625 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:19.841579    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:20.032328 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.032301    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zrpn4"]
Jul 29 00:41:20.034770 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:41:20.034725    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b02ca09_a952_49e2_8df8_40f0d7c716f6.slice/crio-ac8ff0b160d02d0a8dd059811e285d7c4d90b5c5ff60f9afb47c71dd645d25f2 WatchSource:0}: Error finding container ac8ff0b160d02d0a8dd059811e285d7c4d90b5c5ff60f9afb47c71dd645d25f2: Status 404 returned error can't find the container with id ac8ff0b160d02d0a8dd059811e285d7c4d90b5c5ff60f9afb47c71dd645d25f2
Jul 29 00:41:20.510760 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.510724    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4" event={"ID":"9b02ca09-a952-49e2-8df8-40f0d7c716f6","Type":"ContainerStarted","Data":"ac8ff0b160d02d0a8dd059811e285d7c4d90b5c5ff60f9afb47c71dd645d25f2"}
Jul 29 00:41:20.512429 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.512394    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv" event={"ID":"5f2bd4e2-4712-4840-800f-eb424f728dc9","Type":"ContainerStarted","Data":"5a501b28c027dbdac98499af049475c21ecf299e305c6c07be35cb45f8b4dfdb"}
Jul 29 00:41:20.515624 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.515592    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:20.520588 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.520313    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"]
Jul 29 00:41:20.524063 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.524041    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.527615 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.527585    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 29 00:41:20.527615 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.527600    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 00:41:20.527615 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.527605    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 29 00:41:20.527913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.527895    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 29 00:41:20.528067 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.528053    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vgm5s\""
Jul 29 00:41:20.535068 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.535002    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"]
Jul 29 00:41:20.610274 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.610170    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:20.625927 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.625887    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.626122 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.625945    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.626122 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.625977    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.626122 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.626032    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.626317 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.626113    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkn6b\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-kube-api-access-pkn6b\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.626317 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.626167    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43971474-80f0-411c-ad36-1b53b3b35c8c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.626317 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.626213    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.726835 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.726796    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.727026 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.726875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.727026 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.726919    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkn6b\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-kube-api-access-pkn6b\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.727026 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.726972    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43971474-80f0-411c-ad36-1b53b3b35c8c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.727216 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.727041    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.727216 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.727082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.727216 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.727120    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.728419 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.728393    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.732012 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.731987    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.732677 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.732638    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.736038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.736012    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43971474-80f0-411c-ad36-1b53b3b35c8c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.737325 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.737283    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.740071 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.740047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.740687 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.740665    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkn6b\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-kube-api-access-pkn6b\") pod \"istiod-openshift-gateway-7b7995d5cd-lgbns\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:20.838482 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:20.838440    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:21.072564 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:21.072245    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"]
Jul 29 00:41:21.076072 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:41:21.076037    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43971474_80f0_411c_ad36_1b53b3b35c8c.slice/crio-88b5bef96628df617e7c7bea9b222d77a42c40249e1afc96bf4a441ec049f3ba WatchSource:0}: Error finding container 88b5bef96628df617e7c7bea9b222d77a42c40249e1afc96bf4a441ec049f3ba: Status 404 returned error can't find the container with id 88b5bef96628df617e7c7bea9b222d77a42c40249e1afc96bf4a441ec049f3ba
Jul 29 00:41:21.520352 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:21.520307    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns" event={"ID":"43971474-80f0-411c-ad36-1b53b3b35c8c","Type":"ContainerStarted","Data":"88b5bef96628df617e7c7bea9b222d77a42c40249e1afc96bf4a441ec049f3ba"}
Jul 29 00:41:21.601697 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:21.601618    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:25.156923 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.156852    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 00:41:25.157287 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.156982    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 00:41:25.545483 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.545392    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4" event={"ID":"9b02ca09-a952-49e2-8df8-40f0d7c716f6","Type":"ContainerStarted","Data":"ddf4b5aaf65dd77833fe1fb0218523079d4f6a8900d429a0a759f54daeee2ba3"}
Jul 29 00:41:25.545754 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.545730    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:25.547165 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.547142    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv" event={"ID":"5f2bd4e2-4712-4840-800f-eb424f728dc9","Type":"ContainerStarted","Data":"1bf280f61851b70c8cd25ff7a814f91d3db3001f1d3d5aa14018384f1c176606"}
Jul 29 00:41:25.547271 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.547241    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:25.548642 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.548621    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst" event={"ID":"5373cb16-6de6-4fc9-a6b8-a7b7207dc945","Type":"ContainerStarted","Data":"4e75ac5d1c62c5fcf6657b5763159a742ec30300b31309ac29547c2a93152d36"}
Jul 29 00:41:25.550267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.550228    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns" event={"ID":"43971474-80f0-411c-ad36-1b53b3b35c8c","Type":"ContainerStarted","Data":"b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386"}
Jul 29 00:41:25.550368 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.550352    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:25.563884 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.563831    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4" podStartSLOduration=1.514704528 podStartE2EDuration="6.563814586s" totalImagesPullingTime="5.049110058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:41:25.562064162 +0000 UTC m=+315.870992016" watchObservedRunningTime="2026-07-29 00:41:25.563814586 +0000 UTC m=+315.872742427"
Jul 29 00:41:25.582640 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.582577    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns" podStartSLOduration=1.5047914100000002 podStartE2EDuration="5.582557923s" totalImagesPullingTime="4.077766513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:41:25.580145148 +0000 UTC m=+315.889072991" watchObservedRunningTime="2026-07-29 00:41:25.582557923 +0000 UTC m=+315.891485764"
Jul 29 00:41:25.597953 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.597900    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv" podStartSLOduration=2.30864423 podStartE2EDuration="7.597865123s" totalImagesPullingTime="5.289220893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:41:25.59729297 +0000 UTC m=+315.906220813" watchObservedRunningTime="2026-07-29 00:41:25.597865123 +0000 UTC m=+315.906793002"
Jul 29 00:41:25.613748 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.613688    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-75d55f5564-tbqst" podStartSLOduration=1.838762587 podStartE2EDuration="7.61367137s" totalImagesPullingTime="5.774908783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:41:25.613502312 +0000 UTC m=+315.922430155" watchObservedRunningTime="2026-07-29 00:41:25.61367137 +0000 UTC m=+315.922599212"
Jul 29 00:41:25.961146 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:25.961108    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:26.056849 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:26.056817    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:26.219267 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:26.219156    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:26.396911 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:26.396876    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:26.549640 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:26.549547    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:26.555938 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:26.555911    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:29.156935 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.156899    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"]
Jul 29 00:41:29.160117 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.160097    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.207996 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.207961    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d7d553-2b8c-46cf-9884-840447175377-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.208136 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.207998    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86d7d553-2b8c-46cf-9884-840447175377-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.208136 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.208027    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86d7d553-2b8c-46cf-9884-840447175377-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.208136 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.208104    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86d7d553-2b8c-46cf-9884-840447175377-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.208294 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.208145    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d7d553-2b8c-46cf-9884-840447175377-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.208294 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.208249    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4fh\" (UniqueName: \"kubernetes.io/projected/86d7d553-2b8c-46cf-9884-840447175377-kube-api-access-pc4fh\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.208294 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.208276    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86d7d553-2b8c-46cf-9884-840447175377-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.266319 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.266285    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"]
Jul 29 00:41:29.309273 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.309234    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4fh\" (UniqueName: \"kubernetes.io/projected/86d7d553-2b8c-46cf-9884-840447175377-kube-api-access-pc4fh\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.309273 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.309273    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86d7d553-2b8c-46cf-9884-840447175377-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.309511 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.309307    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d7d553-2b8c-46cf-9884-840447175377-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.309511 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.309335    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86d7d553-2b8c-46cf-9884-840447175377-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.309511 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.309372    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86d7d553-2b8c-46cf-9884-840447175377-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.309511 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.309398    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86d7d553-2b8c-46cf-9884-840447175377-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.309511 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.309442    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d7d553-2b8c-46cf-9884-840447175377-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.310240 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.310213    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86d7d553-2b8c-46cf-9884-840447175377-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.311866 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.311835    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d7d553-2b8c-46cf-9884-840447175377-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.311866 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.311863    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86d7d553-2b8c-46cf-9884-840447175377-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.312074 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.312057    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86d7d553-2b8c-46cf-9884-840447175377-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.312188 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.312159    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86d7d553-2b8c-46cf-9884-840447175377-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.318094 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.318073    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4fh\" (UniqueName: \"kubernetes.io/projected/86d7d553-2b8c-46cf-9884-840447175377-kube-api-access-pc4fh\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.318178 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.318148    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d7d553-2b8c-46cf-9884-840447175377-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tpssf\" (UID: \"86d7d553-2b8c-46cf-9884-840447175377\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.469909 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.469828    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:29.668508 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.668469    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"]
Jul 29 00:41:29.669878 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:41:29.669850    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d7d553_2b8c_46cf_9884_840447175377.slice/crio-78267332545aa2e64b122d39787c10981692d6fc862080f03d05c45a4e7bf341 WatchSource:0}: Error finding container 78267332545aa2e64b122d39787c10981692d6fc862080f03d05c45a4e7bf341: Status 404 returned error can't find the container with id 78267332545aa2e64b122d39787c10981692d6fc862080f03d05c45a4e7bf341
Jul 29 00:41:29.673527 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.673484    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 00:41:29.673614 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:29.673572    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 00:41:30.479734 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:30.479685    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:30.575686 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:30.575655    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf" event={"ID":"86d7d553-2b8c-46cf-9884-840447175377","Type":"ContainerStarted","Data":"1744efeac134a5d2a764e31fdabb1a8adef6e410de54de493def5120a9cd0df8"}
Jul 29 00:41:30.575865 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:30.575707    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf" event={"ID":"86d7d553-2b8c-46cf-9884-840447175377","Type":"ContainerStarted","Data":"78267332545aa2e64b122d39787c10981692d6fc862080f03d05c45a4e7bf341"}
Jul 29 00:41:30.575865 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:30.575735    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:30.577519 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:30.577482    2586 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-tpssf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 00:41:30.577619 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:30.577547    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf" podUID="86d7d553-2b8c-46cf-9884-840447175377" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 00:41:30.598544 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:30.598491    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:30.604152 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:30.604102    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf" podStartSLOduration=1.604086074 podStartE2EDuration="1.604086074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:41:30.602023723 +0000 UTC m=+320.910951567" watchObservedRunningTime="2026-07-29 00:41:30.604086074 +0000 UTC m=+320.913013915"
Jul 29 00:41:31.037987 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:31.037952    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:31.166290 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:31.166260    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:31.266179 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:31.266140    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:31.365559 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:31.365512    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:31.581316 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:31.581285    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tpssf"
Jul 29 00:41:31.658207 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:31.658110    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"]
Jul 29 00:41:31.658366 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:31.658306    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns" podUID="43971474-80f0-411c-ad36-1b53b3b35c8c" containerName="discovery" containerID="cri-o://b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386" gracePeriod=30
Jul 29 00:41:31.901226 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:31.901186    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:32.037750 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.037657    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkn6b\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-kube-api-access-pkn6b\") pod \"43971474-80f0-411c-ad36-1b53b3b35c8c\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") "
Jul 29 00:41:32.037908 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.037760    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-ca-configmap\") pod \"43971474-80f0-411c-ad36-1b53b3b35c8c\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") "
Jul 29 00:41:32.037908 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.037784    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-kubeconfig\") pod \"43971474-80f0-411c-ad36-1b53b3b35c8c\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") "
Jul 29 00:41:32.037908 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.037802    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-dns-cert\") pod \"43971474-80f0-411c-ad36-1b53b3b35c8c\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") "
Jul 29 00:41:32.037908 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.037852    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-cacerts\") pod \"43971474-80f0-411c-ad36-1b53b3b35c8c\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") "
Jul 29 00:41:32.038117 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.037949    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-token\") pod \"43971474-80f0-411c-ad36-1b53b3b35c8c\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") "
Jul 29 00:41:32.038117 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.037987    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43971474-80f0-411c-ad36-1b53b3b35c8c-local-certs\") pod \"43971474-80f0-411c-ad36-1b53b3b35c8c\" (UID: \"43971474-80f0-411c-ad36-1b53b3b35c8c\") "
Jul 29 00:41:32.038394 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.038289    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "43971474-80f0-411c-ad36-1b53b3b35c8c" (UID: "43971474-80f0-411c-ad36-1b53b3b35c8c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:41:32.040593 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.040554    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-kube-api-access-pkn6b" (OuterVolumeSpecName: "kube-api-access-pkn6b") pod "43971474-80f0-411c-ad36-1b53b3b35c8c" (UID: "43971474-80f0-411c-ad36-1b53b3b35c8c"). InnerVolumeSpecName "kube-api-access-pkn6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:41:32.041022 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.040679    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "43971474-80f0-411c-ad36-1b53b3b35c8c" (UID: "43971474-80f0-411c-ad36-1b53b3b35c8c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:41:32.041022 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.040704    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "43971474-80f0-411c-ad36-1b53b3b35c8c" (UID: "43971474-80f0-411c-ad36-1b53b3b35c8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:41:32.041022 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.040727    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "43971474-80f0-411c-ad36-1b53b3b35c8c" (UID: "43971474-80f0-411c-ad36-1b53b3b35c8c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:41:32.041022 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.040977    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-cacerts" (OuterVolumeSpecName: "cacerts") pod "43971474-80f0-411c-ad36-1b53b3b35c8c" (UID: "43971474-80f0-411c-ad36-1b53b3b35c8c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:41:32.041022 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.040980    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43971474-80f0-411c-ad36-1b53b3b35c8c-local-certs" (OuterVolumeSpecName: "local-certs") pod "43971474-80f0-411c-ad36-1b53b3b35c8c" (UID: "43971474-80f0-411c-ad36-1b53b3b35c8c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:41:32.139388 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.139347    2586 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-ca-configmap\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:41:32.139388 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.139380    2586 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-kubeconfig\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:41:32.139388 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.139393    2586 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-csr-dns-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:41:32.139627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.139405    2586 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43971474-80f0-411c-ad36-1b53b3b35c8c-cacerts\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:41:32.139627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.139417    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-istio-token\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:41:32.139627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.139429    2586 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43971474-80f0-411c-ad36-1b53b3b35c8c-local-certs\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:41:32.139627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.139439    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkn6b\" (UniqueName: \"kubernetes.io/projected/43971474-80f0-411c-ad36-1b53b3b35c8c-kube-api-access-pkn6b\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:41:32.586421 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.586389    2586 generic.go:356] "Generic (PLEG): container finished" podID="43971474-80f0-411c-ad36-1b53b3b35c8c" containerID="b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386" exitCode=0
Jul 29 00:41:32.586779 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.586459    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"
Jul 29 00:41:32.586779 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.586471    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns" event={"ID":"43971474-80f0-411c-ad36-1b53b3b35c8c","Type":"ContainerDied","Data":"b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386"}
Jul 29 00:41:32.586779 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.586559    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns" event={"ID":"43971474-80f0-411c-ad36-1b53b3b35c8c","Type":"ContainerDied","Data":"88b5bef96628df617e7c7bea9b222d77a42c40249e1afc96bf4a441ec049f3ba"}
Jul 29 00:41:32.586779 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.586587    2586 scope.go:117] "RemoveContainer" containerID="b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386"
Jul 29 00:41:32.594719 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.594703    2586 scope.go:117] "RemoveContainer" containerID="b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386"
Jul 29 00:41:32.594992 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:41:32.594970    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386\": container with ID starting with b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386 not found: ID does not exist" containerID="b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386"
Jul 29 00:41:32.595058 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.595004    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386"} err="failed to get container status \"b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386\": rpc error: code = NotFound desc = could not find container \"b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386\": container with ID starting with b28409046e4ba0275683ad8b3aefe98335d2ca4593ccb549580944be9421e386 not found: ID does not exist"
Jul 29 00:41:32.615091 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.615067    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"]
Jul 29 00:41:32.633498 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:32.633476    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lgbns"]
Jul 29 00:41:34.221076 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:34.221041    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43971474-80f0-411c-ad36-1b53b3b35c8c" path="/var/lib/kubelet/pods/43971474-80f0-411c-ad36-1b53b3b35c8c/volumes"
Jul 29 00:41:36.079216 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:36.079171    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:36.557169 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:36.557083    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-jbxvv"
Jul 29 00:41:36.560334 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:36.560301    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zrpn4"
Jul 29 00:41:37.931973 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:37.931935    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:41:37.990449 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:41:37.990397    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:19.203020 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.202968    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:19.306478 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.306438    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:19.344134 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.344101    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j7vjk"]
Jul 29 00:42:19.344430 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.344416    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43971474-80f0-411c-ad36-1b53b3b35c8c" containerName="discovery"
Jul 29 00:42:19.344491 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.344431    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="43971474-80f0-411c-ad36-1b53b3b35c8c" containerName="discovery"
Jul 29 00:42:19.344540 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.344528    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="43971474-80f0-411c-ad36-1b53b3b35c8c" containerName="discovery"
Jul 29 00:42:19.347308 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.347277    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk"
Jul 29 00:42:19.350026 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.350005    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 00:42:19.350144 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.350007    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 00:42:19.351086 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.351064    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dn2xw\""
Jul 29 00:42:19.357012 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.356990    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j7vjk"]
Jul 29 00:42:19.389120 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.389087    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:19.434781 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.434744    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzfwv\" (UniqueName: \"kubernetes.io/projected/b6a24499-0d80-4451-90c3-1e946f848778-kube-api-access-vzfwv\") pod \"kuadrant-operator-catalog-j7vjk\" (UID: \"b6a24499-0d80-4451-90c3-1e946f848778\") " pod="kuadrant-system/kuadrant-operator-catalog-j7vjk"
Jul 29 00:42:19.536103 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.536005    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzfwv\" (UniqueName: \"kubernetes.io/projected/b6a24499-0d80-4451-90c3-1e946f848778-kube-api-access-vzfwv\") pod \"kuadrant-operator-catalog-j7vjk\" (UID: \"b6a24499-0d80-4451-90c3-1e946f848778\") " pod="kuadrant-system/kuadrant-operator-catalog-j7vjk"
Jul 29 00:42:19.544471 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.544441    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzfwv\" (UniqueName: \"kubernetes.io/projected/b6a24499-0d80-4451-90c3-1e946f848778-kube-api-access-vzfwv\") pod \"kuadrant-operator-catalog-j7vjk\" (UID: \"b6a24499-0d80-4451-90c3-1e946f848778\") " pod="kuadrant-system/kuadrant-operator-catalog-j7vjk"
Jul 29 00:42:19.658603 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.658565    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk"
Jul 29 00:42:19.701346 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.701316    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j7vjk"]
Jul 29 00:42:19.785046 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.785020    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j7vjk"]
Jul 29 00:42:19.787302 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:19.787237    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a24499_0d80_4451_90c3_1e946f848778.slice/crio-6c469646dd35a33e345d145addc9769888d8ec86477df0a382a40a88c9130d13 WatchSource:0}: Error finding container 6c469646dd35a33e345d145addc9769888d8ec86477df0a382a40a88c9130d13: Status 404 returned error can't find the container with id 6c469646dd35a33e345d145addc9769888d8ec86477df0a382a40a88c9130d13
Jul 29 00:42:19.805948 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.805919    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk" event={"ID":"b6a24499-0d80-4451-90c3-1e946f848778","Type":"ContainerStarted","Data":"6c469646dd35a33e345d145addc9769888d8ec86477df0a382a40a88c9130d13"}
Jul 29 00:42:19.906710 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.906672    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5qwr9"]
Jul 29 00:42:19.909370 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.909354    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5qwr9"
Jul 29 00:42:19.917796 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:19.917771    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5qwr9"]
Jul 29 00:42:20.039784 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:20.039697    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nkn\" (UniqueName: \"kubernetes.io/projected/82308f1b-a52a-4f42-ba29-8d3dbbdd454e-kube-api-access-v4nkn\") pod \"kuadrant-operator-catalog-5qwr9\" (UID: \"82308f1b-a52a-4f42-ba29-8d3dbbdd454e\") " pod="kuadrant-system/kuadrant-operator-catalog-5qwr9"
Jul 29 00:42:20.140163 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:20.140121    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nkn\" (UniqueName: \"kubernetes.io/projected/82308f1b-a52a-4f42-ba29-8d3dbbdd454e-kube-api-access-v4nkn\") pod \"kuadrant-operator-catalog-5qwr9\" (UID: \"82308f1b-a52a-4f42-ba29-8d3dbbdd454e\") " pod="kuadrant-system/kuadrant-operator-catalog-5qwr9"
Jul 29 00:42:20.148473 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:20.148431    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nkn\" (UniqueName: \"kubernetes.io/projected/82308f1b-a52a-4f42-ba29-8d3dbbdd454e-kube-api-access-v4nkn\") pod \"kuadrant-operator-catalog-5qwr9\" (UID: \"82308f1b-a52a-4f42-ba29-8d3dbbdd454e\") " pod="kuadrant-system/kuadrant-operator-catalog-5qwr9"
Jul 29 00:42:20.219058 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:20.219025    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5qwr9"
Jul 29 00:42:20.348308 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:20.348284    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5qwr9"]
Jul 29 00:42:20.350652 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:20.350626    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82308f1b_a52a_4f42_ba29_8d3dbbdd454e.slice/crio-d8d3b62690abe42b5fde7267230b82cce2df4ba2e35742837384e0e3e1edc99c WatchSource:0}: Error finding container d8d3b62690abe42b5fde7267230b82cce2df4ba2e35742837384e0e3e1edc99c: Status 404 returned error can't find the container with id d8d3b62690abe42b5fde7267230b82cce2df4ba2e35742837384e0e3e1edc99c
Jul 29 00:42:20.811493 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:20.811462    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5qwr9" event={"ID":"82308f1b-a52a-4f42-ba29-8d3dbbdd454e","Type":"ContainerStarted","Data":"d8d3b62690abe42b5fde7267230b82cce2df4ba2e35742837384e0e3e1edc99c"}
Jul 29 00:42:22.822808 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:22.822769    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk" event={"ID":"b6a24499-0d80-4451-90c3-1e946f848778","Type":"ContainerStarted","Data":"a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282"}
Jul 29 00:42:22.823277 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:22.822805    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk" podUID="b6a24499-0d80-4451-90c3-1e946f848778" containerName="registry-server" containerID="cri-o://a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282" gracePeriod=2
Jul 29 00:42:22.824357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:22.824332    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5qwr9" event={"ID":"82308f1b-a52a-4f42-ba29-8d3dbbdd454e","Type":"ContainerStarted","Data":"f0d03f4f378cc17bdda8095bb2480c092d7f93720e75b735cadae7bc4c1bcbac"}
Jul 29 00:42:22.838032 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:22.837990    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk" podStartSLOduration=1.671557926 podStartE2EDuration="3.837975451s" totalImagesPullingTime="2.166417525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:42:22.836990256 +0000 UTC m=+373.145918099" watchObservedRunningTime="2026-07-29 00:42:22.837975451 +0000 UTC m=+373.146903328"
Jul 29 00:42:22.851559 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:22.851510    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5qwr9" podStartSLOduration=2.245820863 podStartE2EDuration="3.851493299s" totalImagesPullingTime="1.605672436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:42:22.850985881 +0000 UTC m=+373.159913724" watchObservedRunningTime="2026-07-29 00:42:22.851493299 +0000 UTC m=+373.160421141"
Jul 29 00:42:23.067594 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.067566    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk"
Jul 29 00:42:23.167129 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.167027    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzfwv\" (UniqueName: \"kubernetes.io/projected/b6a24499-0d80-4451-90c3-1e946f848778-kube-api-access-vzfwv\") pod \"b6a24499-0d80-4451-90c3-1e946f848778\" (UID: \"b6a24499-0d80-4451-90c3-1e946f848778\") "
Jul 29 00:42:23.169222 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.169185    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a24499-0d80-4451-90c3-1e946f848778-kube-api-access-vzfwv" (OuterVolumeSpecName: "kube-api-access-vzfwv") pod "b6a24499-0d80-4451-90c3-1e946f848778" (UID: "b6a24499-0d80-4451-90c3-1e946f848778"). InnerVolumeSpecName "kube-api-access-vzfwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:42:23.267666 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.267606    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzfwv\" (UniqueName: \"kubernetes.io/projected/b6a24499-0d80-4451-90c3-1e946f848778-kube-api-access-vzfwv\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:23.830375 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.830339    2586 generic.go:356] "Generic (PLEG): container finished" podID="b6a24499-0d80-4451-90c3-1e946f848778" containerID="a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282" exitCode=0
Jul 29 00:42:23.830856 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.830389    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk"
Jul 29 00:42:23.830856 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.830428    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk" event={"ID":"b6a24499-0d80-4451-90c3-1e946f848778","Type":"ContainerDied","Data":"a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282"}
Jul 29 00:42:23.830856 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.830472    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j7vjk" event={"ID":"b6a24499-0d80-4451-90c3-1e946f848778","Type":"ContainerDied","Data":"6c469646dd35a33e345d145addc9769888d8ec86477df0a382a40a88c9130d13"}
Jul 29 00:42:23.830856 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.830493    2586 scope.go:117] "RemoveContainer" containerID="a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282"
Jul 29 00:42:23.839591 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.839576    2586 scope.go:117] "RemoveContainer" containerID="a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282"
Jul 29 00:42:23.839834 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:42:23.839815    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282\": container with ID starting with a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282 not found: ID does not exist" containerID="a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282"
Jul 29 00:42:23.839904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.839842    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282"} err="failed to get container status \"a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282\": rpc error: code = NotFound desc = could not find container \"a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282\": container with ID starting with a04160e297719a1ebf872d6086529e6af80f4a9f10558c4504a73e681b8c2282 not found: ID does not exist"
Jul 29 00:42:23.857074 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.857039    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j7vjk"]
Jul 29 00:42:23.860431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:23.860400    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j7vjk"]
Jul 29 00:42:24.221131 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:24.221040    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a24499-0d80-4451-90c3-1e946f848778" path="/var/lib/kubelet/pods/b6a24499-0d80-4451-90c3-1e946f848778/volumes"
Jul 29 00:42:30.221669 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:30.221641    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-5qwr9"
Jul 29 00:42:30.222110 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:30.221695    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-5qwr9"
Jul 29 00:42:30.241824 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:30.241797    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-5qwr9"
Jul 29 00:42:30.881571 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:30.881543    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-5qwr9"
Jul 29 00:42:34.672109 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.672065    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"]
Jul 29 00:42:34.672591 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.672574    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a24499-0d80-4451-90c3-1e946f848778" containerName="registry-server"
Jul 29 00:42:34.672672 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.672596    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a24499-0d80-4451-90c3-1e946f848778" containerName="registry-server"
Jul 29 00:42:34.672727 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.672689    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a24499-0d80-4451-90c3-1e946f848778" containerName="registry-server"
Jul 29 00:42:34.675955 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.675935    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.678728 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.678708    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-frtpm\""
Jul 29 00:42:34.684803 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.684779    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"]
Jul 29 00:42:34.766317 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.766282    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.766502 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.766323    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv8ll\" (UniqueName: \"kubernetes.io/projected/e6188400-01f0-4993-baef-a71a9a5dfdee-kube-api-access-sv8ll\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.766502 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.766427    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.867863 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.867816    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.867863 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.867864    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv8ll\" (UniqueName: \"kubernetes.io/projected/e6188400-01f0-4993-baef-a71a9a5dfdee-kube-api-access-sv8ll\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.868079 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.867916    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.868256 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.868236    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.868304 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.868262    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.877141 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.877111    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv8ll\" (UniqueName: \"kubernetes.io/projected/e6188400-01f0-4993-baef-a71a9a5dfdee-kube-api-access-sv8ll\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:34.986316 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:34.986219    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:35.071688 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.071659    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"]
Jul 29 00:42:35.085917 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.085841    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"]
Jul 29 00:42:35.086089 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.086032    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.130161 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.130126    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"]
Jul 29 00:42:35.133571 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:35.133544    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6188400_01f0_4993_baef_a71a9a5dfdee.slice/crio-b46b81ea77ff848e74e9de0aa8d4b2e435ead03e1062839df1868d2710fd2927 WatchSource:0}: Error finding container b46b81ea77ff848e74e9de0aa8d4b2e435ead03e1062839df1868d2710fd2927: Status 404 returned error can't find the container with id b46b81ea77ff848e74e9de0aa8d4b2e435ead03e1062839df1868d2710fd2927
Jul 29 00:42:35.170138 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.170109    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.170286 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.170169    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.170286 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.170213    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwlk\" (UniqueName: \"kubernetes.io/projected/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-kube-api-access-pgwlk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.271251 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.271149    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.271251 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.271223    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.271251 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.271242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwlk\" (UniqueName: \"kubernetes.io/projected/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-kube-api-access-pgwlk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.271643 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.271616    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.271681 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.271623    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.280258 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.280222    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwlk\" (UniqueName: \"kubernetes.io/projected/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-kube-api-access-pgwlk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.398012 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.397969    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:35.531528 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.531499    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"]
Jul 29 00:42:35.533513 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:35.533478    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba67441_b9a7_439f_ad86_c3ffecf6af5c.slice/crio-8265a2dee7dcd51243a520a9e1c985077370abbd1a72b1d48eedd6ab952532cd WatchSource:0}: Error finding container 8265a2dee7dcd51243a520a9e1c985077370abbd1a72b1d48eedd6ab952532cd: Status 404 returned error can't find the container with id 8265a2dee7dcd51243a520a9e1c985077370abbd1a72b1d48eedd6ab952532cd
Jul 29 00:42:35.671691 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.671654    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"]
Jul 29 00:42:35.675212 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.675173    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.683179 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.683157    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"]
Jul 29 00:42:35.776344 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.776298    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.776538 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.776384    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.776538 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.776440    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzcwb\" (UniqueName: \"kubernetes.io/projected/96281e7d-5ebb-4387-816c-54bf9cf67b0e-kube-api-access-pzcwb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.877768 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.877733    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcwb\" (UniqueName: \"kubernetes.io/projected/96281e7d-5ebb-4387-816c-54bf9cf67b0e-kube-api-access-pzcwb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.877768 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.877773    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.877979 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.877833    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.878314 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.878292    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.878375 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.878304    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.886351 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.886278    2586 generic.go:356] "Generic (PLEG): container finished" podID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerID="15104bcb5e21df2cb21b0daf457cc30cd579720a98201256e19a0df7e47dce16" exitCode=0
Jul 29 00:42:35.886488 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.886373    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m" event={"ID":"5ba67441-b9a7-439f-ad86-c3ffecf6af5c","Type":"ContainerDied","Data":"15104bcb5e21df2cb21b0daf457cc30cd579720a98201256e19a0df7e47dce16"}
Jul 29 00:42:35.886488 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.886450    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m" event={"ID":"5ba67441-b9a7-439f-ad86-c3ffecf6af5c","Type":"ContainerStarted","Data":"8265a2dee7dcd51243a520a9e1c985077370abbd1a72b1d48eedd6ab952532cd"}
Jul 29 00:42:35.886865 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.886845    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcwb\" (UniqueName: \"kubernetes.io/projected/96281e7d-5ebb-4387-816c-54bf9cf67b0e-kube-api-access-pzcwb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:35.888412 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.888366    2586 generic.go:356] "Generic (PLEG): container finished" podID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerID="fdfbb531c0eacf9fc2159128f559011b2553931b305c26f4d8fc702b75f51592" exitCode=0
Jul 29 00:42:35.888515 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.888407    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm" event={"ID":"e6188400-01f0-4993-baef-a71a9a5dfdee","Type":"ContainerDied","Data":"fdfbb531c0eacf9fc2159128f559011b2553931b305c26f4d8fc702b75f51592"}
Jul 29 00:42:35.888515 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.888466    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm" event={"ID":"e6188400-01f0-4993-baef-a71a9a5dfdee","Type":"ContainerStarted","Data":"b46b81ea77ff848e74e9de0aa8d4b2e435ead03e1062839df1868d2710fd2927"}
Jul 29 00:42:35.996473 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:35.996436    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:36.130275 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.130249    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"]
Jul 29 00:42:36.132601 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:36.132568    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96281e7d_5ebb_4387_816c_54bf9cf67b0e.slice/crio-d00c533ab43193110827556156acad8768181b187048793aafb9f13cb98cd190 WatchSource:0}: Error finding container d00c533ab43193110827556156acad8768181b187048793aafb9f13cb98cd190: Status 404 returned error can't find the container with id d00c533ab43193110827556156acad8768181b187048793aafb9f13cb98cd190
Jul 29 00:42:36.278482 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.278444    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"]
Jul 29 00:42:36.281908 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.281874    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.290987 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.290955    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"]
Jul 29 00:42:36.383271 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.383148    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.383271 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.383230    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgx6\" (UniqueName: \"kubernetes.io/projected/27ecacce-22bf-4746-923c-20d2bd2c8e81-kube-api-access-ffgx6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.383479 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.383325    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.483776 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.483739    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.483958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.483810    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.483958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.483851    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgx6\" (UniqueName: \"kubernetes.io/projected/27ecacce-22bf-4746-923c-20d2bd2c8e81-kube-api-access-ffgx6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.484098 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.484077    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.484222 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.484180    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.493127 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.493101    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgx6\" (UniqueName: \"kubernetes.io/projected/27ecacce-22bf-4746-923c-20d2bd2c8e81-kube-api-access-ffgx6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.592132 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.592091    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:36.733748 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.733720    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"]
Jul 29 00:42:36.776503 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:36.776460    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ecacce_22bf_4746_923c_20d2bd2c8e81.slice/crio-72db2edffa0735ba51382f2ffecf4d33a964a56e99d08826d29ec4dce025e98b WatchSource:0}: Error finding container 72db2edffa0735ba51382f2ffecf4d33a964a56e99d08826d29ec4dce025e98b: Status 404 returned error can't find the container with id 72db2edffa0735ba51382f2ffecf4d33a964a56e99d08826d29ec4dce025e98b
Jul 29 00:42:36.895026 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.894953    2586 generic.go:356] "Generic (PLEG): container finished" podID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerID="a595d6da1dc179e080f6edeb500757b4888c27116e38fcfe281c429d6289430d" exitCode=0
Jul 29 00:42:36.895168 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.895034    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf" event={"ID":"96281e7d-5ebb-4387-816c-54bf9cf67b0e","Type":"ContainerDied","Data":"a595d6da1dc179e080f6edeb500757b4888c27116e38fcfe281c429d6289430d"}
Jul 29 00:42:36.895168 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.895087    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf" event={"ID":"96281e7d-5ebb-4387-816c-54bf9cf67b0e","Type":"ContainerStarted","Data":"d00c533ab43193110827556156acad8768181b187048793aafb9f13cb98cd190"}
Jul 29 00:42:36.897186 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.897156    2586 generic.go:356] "Generic (PLEG): container finished" podID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerID="3c33ffc7bddfdac8f06e289067a39fd4b20b397a954c5c204e73efe2023bae02" exitCode=0
Jul 29 00:42:36.897314 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.897250    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6" event={"ID":"27ecacce-22bf-4746-923c-20d2bd2c8e81","Type":"ContainerDied","Data":"3c33ffc7bddfdac8f06e289067a39fd4b20b397a954c5c204e73efe2023bae02"}
Jul 29 00:42:36.897375 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.897319    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6" event={"ID":"27ecacce-22bf-4746-923c-20d2bd2c8e81","Type":"ContainerStarted","Data":"72db2edffa0735ba51382f2ffecf4d33a964a56e99d08826d29ec4dce025e98b"}
Jul 29 00:42:36.899623 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.899546    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m" event={"ID":"5ba67441-b9a7-439f-ad86-c3ffecf6af5c","Type":"ContainerStarted","Data":"03d3ffea1512d84e69397bfb83a713bf943e08dd32926af7f085722dd96c89e7"}
Jul 29 00:42:36.901910 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.901886    2586 generic.go:356] "Generic (PLEG): container finished" podID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerID="17e0b96314b34d74a6fca543add16c68876a169c55ece06672c8f7d2c003cb34" exitCode=0
Jul 29 00:42:36.902225 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:36.902131    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm" event={"ID":"e6188400-01f0-4993-baef-a71a9a5dfdee","Type":"ContainerDied","Data":"17e0b96314b34d74a6fca543add16c68876a169c55ece06672c8f7d2c003cb34"}
Jul 29 00:42:37.908894 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:37.908848    2586 generic.go:356] "Generic (PLEG): container finished" podID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerID="d85ca4bde254e533c004da2559f383b7c519546a9cb999bd14914a79bf6849c1" exitCode=0
Jul 29 00:42:37.909305 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:37.908934    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm" event={"ID":"e6188400-01f0-4993-baef-a71a9a5dfdee","Type":"ContainerDied","Data":"d85ca4bde254e533c004da2559f383b7c519546a9cb999bd14914a79bf6849c1"}
Jul 29 00:42:37.910696 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:37.910670    2586 generic.go:356] "Generic (PLEG): container finished" podID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerID="1611f11447f69f5f7f4f610d8389c2f6338f08bae836742751a3653c130e381b" exitCode=0
Jul 29 00:42:37.910818 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:37.910746    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf" event={"ID":"96281e7d-5ebb-4387-816c-54bf9cf67b0e","Type":"ContainerDied","Data":"1611f11447f69f5f7f4f610d8389c2f6338f08bae836742751a3653c130e381b"}
Jul 29 00:42:37.912552 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:37.912534    2586 generic.go:356] "Generic (PLEG): container finished" podID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerID="67894bec18703f1667c9cc8b10a50afc490c3d9b019e3dc51c327b06ce1b5881" exitCode=0
Jul 29 00:42:37.912668 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:37.912642    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6" event={"ID":"27ecacce-22bf-4746-923c-20d2bd2c8e81","Type":"ContainerDied","Data":"67894bec18703f1667c9cc8b10a50afc490c3d9b019e3dc51c327b06ce1b5881"}
Jul 29 00:42:37.914681 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:37.914662    2586 generic.go:356] "Generic (PLEG): container finished" podID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerID="03d3ffea1512d84e69397bfb83a713bf943e08dd32926af7f085722dd96c89e7" exitCode=0
Jul 29 00:42:37.914775 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:37.914688    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m" event={"ID":"5ba67441-b9a7-439f-ad86-c3ffecf6af5c","Type":"ContainerDied","Data":"03d3ffea1512d84e69397bfb83a713bf943e08dd32926af7f085722dd96c89e7"}
Jul 29 00:42:38.922489 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:38.922456    2586 generic.go:356] "Generic (PLEG): container finished" podID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerID="e1820d551e0dedf814e5ced060183d5611a94c38f334eee81aa36ad9a8c3deeb" exitCode=0
Jul 29 00:42:38.922829 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:38.922546    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf" event={"ID":"96281e7d-5ebb-4387-816c-54bf9cf67b0e","Type":"ContainerDied","Data":"e1820d551e0dedf814e5ced060183d5611a94c38f334eee81aa36ad9a8c3deeb"}
Jul 29 00:42:38.924436 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:38.924409    2586 generic.go:356] "Generic (PLEG): container finished" podID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerID="e9e3f713036904809e08274e5daafd15b394e5df79a48dc0ab70f48190eb428c" exitCode=0
Jul 29 00:42:38.924545 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:38.924487    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6" event={"ID":"27ecacce-22bf-4746-923c-20d2bd2c8e81","Type":"ContainerDied","Data":"e9e3f713036904809e08274e5daafd15b394e5df79a48dc0ab70f48190eb428c"}
Jul 29 00:42:38.926238 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:38.926219    2586 generic.go:356] "Generic (PLEG): container finished" podID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerID="73018e8a85250ad0b14018b1c5c2183fdc17a2f1b3ba730e41daeeb360606642" exitCode=0
Jul 29 00:42:38.926321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:38.926302    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m" event={"ID":"5ba67441-b9a7-439f-ad86-c3ffecf6af5c","Type":"ContainerDied","Data":"73018e8a85250ad0b14018b1c5c2183fdc17a2f1b3ba730e41daeeb360606642"}
Jul 29 00:42:39.057041 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.057008    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:39.209708 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.209624    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-bundle\") pod \"e6188400-01f0-4993-baef-a71a9a5dfdee\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") "
Jul 29 00:42:39.209708 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.209683    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv8ll\" (UniqueName: \"kubernetes.io/projected/e6188400-01f0-4993-baef-a71a9a5dfdee-kube-api-access-sv8ll\") pod \"e6188400-01f0-4993-baef-a71a9a5dfdee\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") "
Jul 29 00:42:39.209899 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.209758    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-util\") pod \"e6188400-01f0-4993-baef-a71a9a5dfdee\" (UID: \"e6188400-01f0-4993-baef-a71a9a5dfdee\") "
Jul 29 00:42:39.210109 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.210083    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-bundle" (OuterVolumeSpecName: "bundle") pod "e6188400-01f0-4993-baef-a71a9a5dfdee" (UID: "e6188400-01f0-4993-baef-a71a9a5dfdee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:42:39.211808 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.211783    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6188400-01f0-4993-baef-a71a9a5dfdee-kube-api-access-sv8ll" (OuterVolumeSpecName: "kube-api-access-sv8ll") pod "e6188400-01f0-4993-baef-a71a9a5dfdee" (UID: "e6188400-01f0-4993-baef-a71a9a5dfdee"). InnerVolumeSpecName "kube-api-access-sv8ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:42:39.215297 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.215277    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-util" (OuterVolumeSpecName: "util") pod "e6188400-01f0-4993-baef-a71a9a5dfdee" (UID: "e6188400-01f0-4993-baef-a71a9a5dfdee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:42:39.311063 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.311026    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv8ll\" (UniqueName: \"kubernetes.io/projected/e6188400-01f0-4993-baef-a71a9a5dfdee-kube-api-access-sv8ll\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:39.311063 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.311055    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-util\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:39.311063 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.311066    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6188400-01f0-4993-baef-a71a9a5dfdee-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:39.933317 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.933281    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm"
Jul 29 00:42:39.933317 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.933293    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm" event={"ID":"e6188400-01f0-4993-baef-a71a9a5dfdee","Type":"ContainerDied","Data":"b46b81ea77ff848e74e9de0aa8d4b2e435ead03e1062839df1868d2710fd2927"}
Jul 29 00:42:39.933892 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:39.933347    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b46b81ea77ff848e74e9de0aa8d4b2e435ead03e1062839df1868d2710fd2927"
Jul 29 00:42:40.093836 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.093809    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:40.114779 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.114754    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:40.118666 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.118640    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:40.218177 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.218083    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-bundle\") pod \"27ecacce-22bf-4746-923c-20d2bd2c8e81\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") "
Jul 29 00:42:40.218177 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.218127    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-bundle\") pod \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") "
Jul 29 00:42:40.218177 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.218167    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-util\") pod \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") "
Jul 29 00:42:40.218489 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.218219    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-bundle\") pod \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") "
Jul 29 00:42:40.218489 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.218263    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzcwb\" (UniqueName: \"kubernetes.io/projected/96281e7d-5ebb-4387-816c-54bf9cf67b0e-kube-api-access-pzcwb\") pod \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") "
Jul 29 00:42:40.218489 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.218299    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffgx6\" (UniqueName: \"kubernetes.io/projected/27ecacce-22bf-4746-923c-20d2bd2c8e81-kube-api-access-ffgx6\") pod \"27ecacce-22bf-4746-923c-20d2bd2c8e81\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") "
Jul 29 00:42:40.218489 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.218351    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgwlk\" (UniqueName: \"kubernetes.io/projected/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-kube-api-access-pgwlk\") pod \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\" (UID: \"5ba67441-b9a7-439f-ad86-c3ffecf6af5c\") "
Jul 29 00:42:40.218839 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.218780    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-bundle" (OuterVolumeSpecName: "bundle") pod "96281e7d-5ebb-4387-816c-54bf9cf67b0e" (UID: "96281e7d-5ebb-4387-816c-54bf9cf67b0e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:42:40.218923 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.218852    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-bundle" (OuterVolumeSpecName: "bundle") pod "27ecacce-22bf-4746-923c-20d2bd2c8e81" (UID: "27ecacce-22bf-4746-923c-20d2bd2c8e81"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:42:40.219463 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.219435    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-util\") pod \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\" (UID: \"96281e7d-5ebb-4387-816c-54bf9cf67b0e\") "
Jul 29 00:42:40.219986 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.219959    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-bundle" (OuterVolumeSpecName: "bundle") pod "5ba67441-b9a7-439f-ad86-c3ffecf6af5c" (UID: "5ba67441-b9a7-439f-ad86-c3ffecf6af5c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:42:40.220089 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.220038    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-util\") pod \"27ecacce-22bf-4746-923c-20d2bd2c8e81\" (UID: \"27ecacce-22bf-4746-923c-20d2bd2c8e81\") "
Jul 29 00:42:40.221470 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.221444    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96281e7d-5ebb-4387-816c-54bf9cf67b0e-kube-api-access-pzcwb" (OuterVolumeSpecName: "kube-api-access-pzcwb") pod "96281e7d-5ebb-4387-816c-54bf9cf67b0e" (UID: "96281e7d-5ebb-4387-816c-54bf9cf67b0e"). InnerVolumeSpecName "kube-api-access-pzcwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:42:40.221627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.221600    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ecacce-22bf-4746-923c-20d2bd2c8e81-kube-api-access-ffgx6" (OuterVolumeSpecName: "kube-api-access-ffgx6") pod "27ecacce-22bf-4746-923c-20d2bd2c8e81" (UID: "27ecacce-22bf-4746-923c-20d2bd2c8e81"). InnerVolumeSpecName "kube-api-access-ffgx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:42:40.222186 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.222158    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-kube-api-access-pgwlk" (OuterVolumeSpecName: "kube-api-access-pgwlk") pod "5ba67441-b9a7-439f-ad86-c3ffecf6af5c" (UID: "5ba67441-b9a7-439f-ad86-c3ffecf6af5c"). InnerVolumeSpecName "kube-api-access-pgwlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:42:40.223405 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.223340    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzcwb\" (UniqueName: \"kubernetes.io/projected/96281e7d-5ebb-4387-816c-54bf9cf67b0e-kube-api-access-pzcwb\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:40.223405 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.223369    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgx6\" (UniqueName: \"kubernetes.io/projected/27ecacce-22bf-4746-923c-20d2bd2c8e81-kube-api-access-ffgx6\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:40.223405 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.223383    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgwlk\" (UniqueName: \"kubernetes.io/projected/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-kube-api-access-pgwlk\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:40.223405 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.223398    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:40.223663 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.223412    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:40.223663 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.223424    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:40.227255 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.227225    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-util" (OuterVolumeSpecName: "util") pod "5ba67441-b9a7-439f-ad86-c3ffecf6af5c" (UID: "5ba67441-b9a7-439f-ad86-c3ffecf6af5c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:42:40.229820 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.229792    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-util" (OuterVolumeSpecName: "util") pod "27ecacce-22bf-4746-923c-20d2bd2c8e81" (UID: "27ecacce-22bf-4746-923c-20d2bd2c8e81"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:42:40.229820 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.229805    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-util" (OuterVolumeSpecName: "util") pod "96281e7d-5ebb-4387-816c-54bf9cf67b0e" (UID: "96281e7d-5ebb-4387-816c-54bf9cf67b0e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:42:40.324163 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.324119    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ba67441-b9a7-439f-ad86-c3ffecf6af5c-util\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:40.324163 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.324153    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96281e7d-5ebb-4387-816c-54bf9cf67b0e-util\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:40.324163 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.324161    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27ecacce-22bf-4746-923c-20d2bd2c8e81-util\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:42:40.939596 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.939567    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf"
Jul 29 00:42:40.940030 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.939567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf" event={"ID":"96281e7d-5ebb-4387-816c-54bf9cf67b0e","Type":"ContainerDied","Data":"d00c533ab43193110827556156acad8768181b187048793aafb9f13cb98cd190"}
Jul 29 00:42:40.940030 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.939688    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d00c533ab43193110827556156acad8768181b187048793aafb9f13cb98cd190"
Jul 29 00:42:40.941558 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.941538    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6"
Jul 29 00:42:40.941679 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.941567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6" event={"ID":"27ecacce-22bf-4746-923c-20d2bd2c8e81","Type":"ContainerDied","Data":"72db2edffa0735ba51382f2ffecf4d33a964a56e99d08826d29ec4dce025e98b"}
Jul 29 00:42:40.941679 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.941607    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72db2edffa0735ba51382f2ffecf4d33a964a56e99d08826d29ec4dce025e98b"
Jul 29 00:42:40.943607 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.943589    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m"
Jul 29 00:42:40.943712 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.943619    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m" event={"ID":"5ba67441-b9a7-439f-ad86-c3ffecf6af5c","Type":"ContainerDied","Data":"8265a2dee7dcd51243a520a9e1c985077370abbd1a72b1d48eedd6ab952532cd"}
Jul 29 00:42:40.943712 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:40.943658    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8265a2dee7dcd51243a520a9e1c985077370abbd1a72b1d48eedd6ab952532cd"
Jul 29 00:42:42.940981 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:42.940951    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:43.547975 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:43.547945    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:43.647385 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:43.647350    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:45.571381 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571342    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"]
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571673    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerName="pull"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571687    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerName="pull"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571696    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerName="util"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571701    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerName="util"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571708    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerName="pull"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571714    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerName="pull"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571720    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerName="extract"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571726    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerName="extract"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571734    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerName="extract"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571739    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerName="extract"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571746    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerName="util"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571750    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerName="util"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571757    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerName="util"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571762    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerName="util"
Jul 29 00:42:45.571765 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571768    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerName="pull"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571773    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerName="pull"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571780    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerName="util"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571785    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerName="util"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571792    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerName="pull"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571797    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerName="pull"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571803    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerName="extract"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571808    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerName="extract"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571815    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerName="extract"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571820    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerName="extract"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571867    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6188400-01f0-4993-baef-a71a9a5dfdee" containerName="extract"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571873    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ecacce-22bf-4746-923c-20d2bd2c8e81" containerName="extract"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571879    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba67441-b9a7-439f-ad86-c3ffecf6af5c" containerName="extract"
Jul 29 00:42:45.572221 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.571884    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="96281e7d-5ebb-4387-816c-54bf9cf67b0e" containerName="extract"
Jul 29 00:42:45.576049 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.576031    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"
Jul 29 00:42:45.579223 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.579183    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 29 00:42:45.579333 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.579186    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cchvf\""
Jul 29 00:42:45.589541 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.589516    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"]
Jul 29 00:42:45.672909 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.672877    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpnkp\" (UniqueName: \"kubernetes.io/projected/9071ce25-d8c5-4666-9b8c-8b528dd5c056-kube-api-access-jpnkp\") pod \"dns-operator-controller-manager-864dd78c4c-7gdvt\" (UID: \"9071ce25-d8c5-4666-9b8c-8b528dd5c056\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"
Jul 29 00:42:45.774007 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.773969    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpnkp\" (UniqueName: \"kubernetes.io/projected/9071ce25-d8c5-4666-9b8c-8b528dd5c056-kube-api-access-jpnkp\") pod \"dns-operator-controller-manager-864dd78c4c-7gdvt\" (UID: \"9071ce25-d8c5-4666-9b8c-8b528dd5c056\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"
Jul 29 00:42:45.783480 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.783442    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpnkp\" (UniqueName: \"kubernetes.io/projected/9071ce25-d8c5-4666-9b8c-8b528dd5c056-kube-api-access-jpnkp\") pod \"dns-operator-controller-manager-864dd78c4c-7gdvt\" (UID: \"9071ce25-d8c5-4666-9b8c-8b528dd5c056\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"
Jul 29 00:42:45.886177 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:45.886088    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"
Jul 29 00:42:46.019711 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:46.019688    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"]
Jul 29 00:42:46.021600 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:46.021568    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9071ce25_d8c5_4666_9b8c_8b528dd5c056.slice/crio-51f08188293207482eb08831cdf5f351e899f01a5f6121b099da70c33d52f605 WatchSource:0}: Error finding container 51f08188293207482eb08831cdf5f351e899f01a5f6121b099da70c33d52f605: Status 404 returned error can't find the container with id 51f08188293207482eb08831cdf5f351e899f01a5f6121b099da70c33d52f605
Jul 29 00:42:46.976934 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:46.976895    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt" event={"ID":"9071ce25-d8c5-4666-9b8c-8b528dd5c056","Type":"ContainerStarted","Data":"51f08188293207482eb08831cdf5f351e899f01a5f6121b099da70c33d52f605"}
Jul 29 00:42:47.200606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.200570    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:47.275547 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.275455    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-778ff5dc99-xj9wj"]
Jul 29 00:42:47.279786 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.279755    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.312791 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.312759    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778ff5dc99-xj9wj"]
Jul 29 00:42:47.350048 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.350008    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:47.387603 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.387560    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e50c874b-77a2-4a2f-a2e4-9fa281363a19-console-oauth-config\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.387603 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.387606    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-trusted-ca-bundle\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.387853 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.387727    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-console-config\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.387853 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.387802    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e50c874b-77a2-4a2f-a2e4-9fa281363a19-console-serving-cert\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.387853 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.387829    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-oauth-serving-cert\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.388015 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.387980    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-service-ca\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.388117 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.388061    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jgn\" (UniqueName: \"kubernetes.io/projected/e50c874b-77a2-4a2f-a2e4-9fa281363a19-kube-api-access-k2jgn\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.447672 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.447640    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:47.489279 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.489236    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jgn\" (UniqueName: \"kubernetes.io/projected/e50c874b-77a2-4a2f-a2e4-9fa281363a19-kube-api-access-k2jgn\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.489452 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.489301    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e50c874b-77a2-4a2f-a2e4-9fa281363a19-console-oauth-config\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.489452 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.489327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-trusted-ca-bundle\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.489452 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.489349    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-console-config\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.489452 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.489378    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e50c874b-77a2-4a2f-a2e4-9fa281363a19-console-serving-cert\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.489452 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.489401    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-oauth-serving-cert\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.489731 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.489455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-service-ca\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.490245 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.490218    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-console-config\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.490364 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.490271    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-service-ca\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.490364 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.490343    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-trusted-ca-bundle\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.490849 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.490832    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e50c874b-77a2-4a2f-a2e4-9fa281363a19-oauth-serving-cert\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.492031 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.492009    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e50c874b-77a2-4a2f-a2e4-9fa281363a19-console-serving-cert\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.492136 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.492115    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e50c874b-77a2-4a2f-a2e4-9fa281363a19-console-oauth-config\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.507184 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.507160    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jgn\" (UniqueName: \"kubernetes.io/projected/e50c874b-77a2-4a2f-a2e4-9fa281363a19-kube-api-access-k2jgn\") pod \"console-778ff5dc99-xj9wj\" (UID: \"e50c874b-77a2-4a2f-a2e4-9fa281363a19\") " pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.592433 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.592393    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:47.902724 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.902700    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778ff5dc99-xj9wj"]
Jul 29 00:42:47.904146 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:47.904120    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50c874b_77a2_4a2f_a2e4_9fa281363a19.slice/crio-3b740228e8e6856a0dd79bbdaac0179d0d048511ac212cb1f087dc38fe4d6758 WatchSource:0}: Error finding container 3b740228e8e6856a0dd79bbdaac0179d0d048511ac212cb1f087dc38fe4d6758: Status 404 returned error can't find the container with id 3b740228e8e6856a0dd79bbdaac0179d0d048511ac212cb1f087dc38fe4d6758
Jul 29 00:42:47.983323 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.983288    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt" event={"ID":"9071ce25-d8c5-4666-9b8c-8b528dd5c056","Type":"ContainerStarted","Data":"c3a91fac44711b9f4f8bb40ed4f27f2552caec3734dcaf67656ff430fde7c15f"}
Jul 29 00:42:47.983709 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.983501    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"
Jul 29 00:42:47.985047 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.985024    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff5dc99-xj9wj" event={"ID":"e50c874b-77a2-4a2f-a2e4-9fa281363a19","Type":"ContainerStarted","Data":"3af3e58c01835f689b41f253b5fef15bd732bd1d94dd13fb087308cd43282ad8"}
Jul 29 00:42:47.985143 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:47.985063    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff5dc99-xj9wj" event={"ID":"e50c874b-77a2-4a2f-a2e4-9fa281363a19","Type":"ContainerStarted","Data":"3b740228e8e6856a0dd79bbdaac0179d0d048511ac212cb1f087dc38fe4d6758"}
Jul 29 00:42:48.005499 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:48.005433    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt" podStartSLOduration=1.204367398 podStartE2EDuration="3.005415341s" totalImagesPullingTime="1.801047943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:42:48.003295252 +0000 UTC m=+398.312223091" watchObservedRunningTime="2026-07-29 00:42:48.005415341 +0000 UTC m=+398.314343184"
Jul 29 00:42:48.021824 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:48.021767    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-778ff5dc99-xj9wj" podStartSLOduration=1.021749924 podStartE2EDuration="1.021749924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:42:48.021016479 +0000 UTC m=+398.329944325" watchObservedRunningTime="2026-07-29 00:42:48.021749924 +0000 UTC m=+398.330677766"
Jul 29 00:42:49.008769 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.008734    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"]
Jul 29 00:42:49.012184 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.012163    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:42:49.015041 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.015014    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-t49dn\""
Jul 29 00:42:49.022708 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.022687    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"]
Jul 29 00:42:49.105343 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.105311    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9sw5\" (UniqueName: \"kubernetes.io/projected/eb9a01a9-0c60-41ff-8824-8cfba76a8160-kube-api-access-m9sw5\") pod \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" (UID: \"eb9a01a9-0c60-41ff-8824-8cfba76a8160\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:42:49.206209 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.206155    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9sw5\" (UniqueName: \"kubernetes.io/projected/eb9a01a9-0c60-41ff-8824-8cfba76a8160-kube-api-access-m9sw5\") pod \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" (UID: \"eb9a01a9-0c60-41ff-8824-8cfba76a8160\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:42:49.218270 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.218231    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9sw5\" (UniqueName: \"kubernetes.io/projected/eb9a01a9-0c60-41ff-8824-8cfba76a8160-kube-api-access-m9sw5\") pod \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" (UID: \"eb9a01a9-0c60-41ff-8824-8cfba76a8160\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:42:49.323260 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.323218    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:42:49.455386 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.455350    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"]
Jul 29 00:42:49.458386 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:49.458357    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9a01a9_0c60_41ff_8824_8cfba76a8160.slice/crio-83e60690ae8bc0745e5e8666f57c5e9f7ff0f768c25e63e380b2ccf8e7f80fca WatchSource:0}: Error finding container 83e60690ae8bc0745e5e8666f57c5e9f7ff0f768c25e63e380b2ccf8e7f80fca: Status 404 returned error can't find the container with id 83e60690ae8bc0745e5e8666f57c5e9f7ff0f768c25e63e380b2ccf8e7f80fca
Jul 29 00:42:49.578274 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.578167    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:49.723652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.723618    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:49.996672 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:49.996596    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9" event={"ID":"eb9a01a9-0c60-41ff-8824-8cfba76a8160","Type":"ContainerStarted","Data":"83e60690ae8bc0745e5e8666f57c5e9f7ff0f768c25e63e380b2ccf8e7f80fca"}
Jul 29 00:42:50.970557 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:50.970519    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:52.009905 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:52.009865    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9" event={"ID":"eb9a01a9-0c60-41ff-8824-8cfba76a8160","Type":"ContainerStarted","Data":"6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd"}
Jul 29 00:42:52.010330 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:52.009984    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:42:52.029556 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:52.029498    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9" podStartSLOduration=2.2582139039999998 podStartE2EDuration="4.029480557s" totalImagesPullingTime="1.771266653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:42:52.02717114 +0000 UTC m=+402.336098983" watchObservedRunningTime="2026-07-29 00:42:52.029480557 +0000 UTC m=+402.338408399"
Jul 29 00:42:53.848541 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:53.848505    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"]
Jul 29 00:42:53.856610 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:53.856587    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:42:53.859359 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:53.859331    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sq5mr\""
Jul 29 00:42:53.864897 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:53.864873    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"]
Jul 29 00:42:53.941153 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:53.941109    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e75683c-cc23-4d4b-b224-97dd7f1b6107-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" (UID: \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:42:53.941153 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:53.941154    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvrq\" (UniqueName: \"kubernetes.io/projected/6e75683c-cc23-4d4b-b224-97dd7f1b6107-kube-api-access-wwvrq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" (UID: \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:42:54.041917 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.041873    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e75683c-cc23-4d4b-b224-97dd7f1b6107-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" (UID: \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:42:54.041917 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.041917    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvrq\" (UniqueName: \"kubernetes.io/projected/6e75683c-cc23-4d4b-b224-97dd7f1b6107-kube-api-access-wwvrq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" (UID: \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:42:54.042361 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.042339    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e75683c-cc23-4d4b-b224-97dd7f1b6107-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" (UID: \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:42:54.053872 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.053843    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvrq\" (UniqueName: \"kubernetes.io/projected/6e75683c-cc23-4d4b-b224-97dd7f1b6107-kube-api-access-wwvrq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" (UID: \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:42:54.136041 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.135954    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:54.167721 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.167692    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:42:54.312890 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.312862    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"]
Jul 29 00:42:54.314742 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:54.314712    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e75683c_cc23_4d4b_b224_97dd7f1b6107.slice/crio-c8a5589ad95b171dcc8996bfc1352677af83d774819b4951a863460065ae8dbd WatchSource:0}: Error finding container c8a5589ad95b171dcc8996bfc1352677af83d774819b4951a863460065ae8dbd: Status 404 returned error can't find the container with id c8a5589ad95b171dcc8996bfc1352677af83d774819b4951a863460065ae8dbd
Jul 29 00:42:54.538771 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.538689    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:54.825734 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.825708    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:54.918917 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:54.918885    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:55.027413 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:55.027377    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk" event={"ID":"6e75683c-cc23-4d4b-b224-97dd7f1b6107","Type":"ContainerStarted","Data":"c8a5589ad95b171dcc8996bfc1352677af83d774819b4951a863460065ae8dbd"}
Jul 29 00:42:57.138843 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.138807    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"]
Jul 29 00:42:57.142522 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.142501    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"
Jul 29 00:42:57.145554 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.145526    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cctzp\""
Jul 29 00:42:57.155090 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.155061    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"]
Jul 29 00:42:57.169308 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.169275    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97q6c\" (UniqueName: \"kubernetes.io/projected/cc62d8dc-1eea-4ef7-b61a-429f104447ae-kube-api-access-97q6c\") pod \"authorino-operator-75cc8d5cb7-bfpgh\" (UID: \"cc62d8dc-1eea-4ef7-b61a-429f104447ae\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"
Jul 29 00:42:57.269789 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.269743    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97q6c\" (UniqueName: \"kubernetes.io/projected/cc62d8dc-1eea-4ef7-b61a-429f104447ae-kube-api-access-97q6c\") pod \"authorino-operator-75cc8d5cb7-bfpgh\" (UID: \"cc62d8dc-1eea-4ef7-b61a-429f104447ae\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"
Jul 29 00:42:57.281441 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.281405    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97q6c\" (UniqueName: \"kubernetes.io/projected/cc62d8dc-1eea-4ef7-b61a-429f104447ae-kube-api-access-97q6c\") pod \"authorino-operator-75cc8d5cb7-bfpgh\" (UID: \"cc62d8dc-1eea-4ef7-b61a-429f104447ae\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"
Jul 29 00:42:57.454951 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.454857    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"
Jul 29 00:42:57.592938 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.592903    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:57.593108 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.592982    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:57.597786 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.597762    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:57.613246 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.613189    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:57.625443 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:57.625352    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"]
Jul 29 00:42:57.628213 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:42:57.628163    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc62d8dc_1eea_4ef7_b61a_429f104447ae.slice/crio-d02ef8ae02e6600687a1a7230e7f3a340b5976fc4fc778f2fc284e1ef9cf4a81 WatchSource:0}: Error finding container d02ef8ae02e6600687a1a7230e7f3a340b5976fc4fc778f2fc284e1ef9cf4a81: Status 404 returned error can't find the container with id d02ef8ae02e6600687a1a7230e7f3a340b5976fc4fc778f2fc284e1ef9cf4a81
Jul 29 00:42:58.048234 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:58.048182    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh" event={"ID":"cc62d8dc-1eea-4ef7-b61a-429f104447ae","Type":"ContainerStarted","Data":"d02ef8ae02e6600687a1a7230e7f3a340b5976fc4fc778f2fc284e1ef9cf4a81"}
Jul 29 00:42:58.052245 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:58.052213    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-778ff5dc99-xj9wj"
Jul 29 00:42:58.104244 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:58.103819    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9fbffb86-gwb6j"]
Jul 29 00:42:58.223110 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:58.223077    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:58.423825 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:58.423765    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:58.523742 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:58.523699    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:58.944734 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:58.944696    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:42:58.993447 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:58.993416    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gdvt"
Jul 29 00:42:59.016651 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:42:59.016613    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:02.074939 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:02.074898    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk" event={"ID":"6e75683c-cc23-4d4b-b224-97dd7f1b6107","Type":"ContainerStarted","Data":"6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227"}
Jul 29 00:43:02.075422 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:02.075079    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:02.076422 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:02.076393    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh" event={"ID":"cc62d8dc-1eea-4ef7-b61a-429f104447ae","Type":"ContainerStarted","Data":"18ca49c9e02aa61dc4f72ee194287fefb9df4f0a1aaa2905a357457b00c195a9"}
Jul 29 00:43:02.076590 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:02.076574    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"
Jul 29 00:43:02.100539 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:02.100469    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk" podStartSLOduration=2.357251588 podStartE2EDuration="9.100449003s" totalImagesPullingTime="6.743197415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:43:02.100128918 +0000 UTC m=+412.409056765" watchObservedRunningTime="2026-07-29 00:43:02.100449003 +0000 UTC m=+412.409376845"
Jul 29 00:43:02.120038 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:02.119988    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh" podStartSLOduration=1.649902567 podStartE2EDuration="5.119969937s" totalImagesPullingTime="3.47006737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:43:02.119682226 +0000 UTC m=+412.428610069" watchObservedRunningTime="2026-07-29 00:43:02.119969937 +0000 UTC m=+412.428897780"
Jul 29 00:43:02.746710 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:02.746677    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:02.846345 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:02.846308    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:03.018513 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:03.018421    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:04.101416 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:04.101385    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:04.366364 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:04.366276    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:06.132366 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:06.132334    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:06.233166 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:06.233132    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:13.084164 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:13.084129    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:13.084684 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:13.084237    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bfpgh"
Jul 29 00:43:14.572370 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.572332    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:14.671154 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.671124    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:14.834431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.834341    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"]
Jul 29 00:43:14.834592 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.834522    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" containerName="manager" containerID="cri-o://6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227" gracePeriod=2
Jul 29 00:43:14.844994 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.844959    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"]
Jul 29 00:43:14.858958 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.858929    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"]
Jul 29 00:43:14.859147 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.859124    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" containerName="manager" containerID="cri-o://6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd" gracePeriod=2
Jul 29 00:43:14.868737 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.868703    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"]
Jul 29 00:43:14.874287 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:14.874255    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:14.891189 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.891158    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"]
Jul 29 00:43:14.891537 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.891522    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" containerName="manager"
Jul 29 00:43:14.891606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.891538    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" containerName="manager"
Jul 29 00:43:14.891606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.891556    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" containerName="manager"
Jul 29 00:43:14.891606 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.891562    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" containerName="manager"
Jul 29 00:43:14.891771 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.891621    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" containerName="manager"
Jul 29 00:43:14.891771 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.891632    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" containerName="manager"
Jul 29 00:43:14.894666 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.894648    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"
Jul 29 00:43:14.897113 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:14.897078    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:14.899049 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:14.899015    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:14.908448 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:14.908372    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"]
Jul 29 00:43:15.035177 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.035141    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgw6b\" (UniqueName: \"kubernetes.io/projected/ee7b1eaf-3df1-49ba-b8af-522596a7e862-kube-api-access-wgw6b\") pod \"limitador-operator-controller-manager-859d7b78bd-lm7xw\" (UID: \"ee7b1eaf-3df1-49ba-b8af-522596a7e862\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"
Jul 29 00:43:15.107648 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.107621    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:15.110289 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.110260    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:15.110425 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.110357    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:15.112300 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.112274    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:15.114319 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.114298    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:15.116355 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.116333    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:15.135760 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.135725    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgw6b\" (UniqueName: \"kubernetes.io/projected/ee7b1eaf-3df1-49ba-b8af-522596a7e862-kube-api-access-wgw6b\") pod \"limitador-operator-controller-manager-859d7b78bd-lm7xw\" (UID: \"ee7b1eaf-3df1-49ba-b8af-522596a7e862\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"
Jul 29 00:43:15.145353 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.145323    2586 generic.go:356] "Generic (PLEG): container finished" podID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" containerID="6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227" exitCode=0
Jul 29 00:43:15.145494 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.145368    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:15.145494 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.145424    2586 scope.go:117] "RemoveContainer" containerID="6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227"
Jul 29 00:43:15.146804 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.146777    2586 generic.go:356] "Generic (PLEG): container finished" podID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" containerID="6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd" exitCode=0
Jul 29 00:43:15.146919 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.146821    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:15.148036 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.148008    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:15.150036 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.150013    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:15.152100 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.152071    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:15.154079 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.154058    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:15.154295 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.154276    2586 scope.go:117] "RemoveContainer" containerID="6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227"
Jul 29 00:43:15.154574 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.154555    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227\": container with ID starting with 6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227 not found: ID does not exist" containerID="6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227"
Jul 29 00:43:15.154638 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.154587    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227"} err="failed to get container status \"6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227\": rpc error: code = NotFound desc = could not find container \"6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227\": container with ID starting with 6f0dd54ea6ea4339e5cc0efea561289e09adc0fa7c44a2debc74dfe552a20227 not found: ID does not exist"
Jul 29 00:43:15.154638 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.154606    2586 scope.go:117] "RemoveContainer" containerID="6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd"
Jul 29 00:43:15.157739 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.157716    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgw6b\" (UniqueName: \"kubernetes.io/projected/ee7b1eaf-3df1-49ba-b8af-522596a7e862-kube-api-access-wgw6b\") pod \"limitador-operator-controller-manager-859d7b78bd-lm7xw\" (UID: \"ee7b1eaf-3df1-49ba-b8af-522596a7e862\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"
Jul 29 00:43:15.162343 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.162326    2586 scope.go:117] "RemoveContainer" containerID="6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd"
Jul 29 00:43:15.162605 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.162590    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd\": container with ID starting with 6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd not found: ID does not exist" containerID="6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd"
Jul 29 00:43:15.162652 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.162612    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd"} err="failed to get container status \"6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd\": rpc error: code = NotFound desc = could not find container \"6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd\": container with ID starting with 6c5f6fd0322956e57dbfefaf0f3e667ceb357fb72a6423bca33a7ac882706afd not found: ID does not exist"
Jul 29 00:43:15.236949 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.236914    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwvrq\" (UniqueName: \"kubernetes.io/projected/6e75683c-cc23-4d4b-b224-97dd7f1b6107-kube-api-access-wwvrq\") pod \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\" (UID: \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\") "
Jul 29 00:43:15.237117 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.237002    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e75683c-cc23-4d4b-b224-97dd7f1b6107-extensions-socket-volume\") pod \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\" (UID: \"6e75683c-cc23-4d4b-b224-97dd7f1b6107\") "
Jul 29 00:43:15.237117 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.237051    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9sw5\" (UniqueName: \"kubernetes.io/projected/eb9a01a9-0c60-41ff-8824-8cfba76a8160-kube-api-access-m9sw5\") pod \"eb9a01a9-0c60-41ff-8824-8cfba76a8160\" (UID: \"eb9a01a9-0c60-41ff-8824-8cfba76a8160\") "
Jul 29 00:43:15.237431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.237404    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e75683c-cc23-4d4b-b224-97dd7f1b6107-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6e75683c-cc23-4d4b-b224-97dd7f1b6107" (UID: "6e75683c-cc23-4d4b-b224-97dd7f1b6107"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:43:15.239103 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.239080    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9a01a9-0c60-41ff-8824-8cfba76a8160-kube-api-access-m9sw5" (OuterVolumeSpecName: "kube-api-access-m9sw5") pod "eb9a01a9-0c60-41ff-8824-8cfba76a8160" (UID: "eb9a01a9-0c60-41ff-8824-8cfba76a8160"). InnerVolumeSpecName "kube-api-access-m9sw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:43:15.239103 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.239096    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e75683c-cc23-4d4b-b224-97dd7f1b6107-kube-api-access-wwvrq" (OuterVolumeSpecName: "kube-api-access-wwvrq") pod "6e75683c-cc23-4d4b-b224-97dd7f1b6107" (UID: "6e75683c-cc23-4d4b-b224-97dd7f1b6107"). InnerVolumeSpecName "kube-api-access-wwvrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:43:15.280971 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.280936    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"
Jul 29 00:43:15.338487 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.338049    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9sw5\" (UniqueName: \"kubernetes.io/projected/eb9a01a9-0c60-41ff-8824-8cfba76a8160-kube-api-access-m9sw5\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:15.338487 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.338087    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwvrq\" (UniqueName: \"kubernetes.io/projected/6e75683c-cc23-4d4b-b224-97dd7f1b6107-kube-api-access-wwvrq\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:15.338487 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.338172    2586 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e75683c-cc23-4d4b-b224-97dd7f1b6107-extensions-socket-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:15.418875 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:15.418844    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"]
Jul 29 00:43:15.421772 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:43:15.421744    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7b1eaf_3df1_49ba_b8af_522596a7e862.slice/crio-e1495230515253ad7002a26479e9393cf6d37e0437734a00cd4973d8a2bcc326 WatchSource:0}: Error finding container e1495230515253ad7002a26479e9393cf6d37e0437734a00cd4973d8a2bcc326: Status 404 returned error can't find the container with id e1495230515253ad7002a26479e9393cf6d37e0437734a00cd4973d8a2bcc326
Jul 29 00:43:15.459777 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.459743    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:15.461626 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.461592    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:15.463480 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.463453    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:15.465394 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:15.465367    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:16.153921 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:16.153812    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw" event={"ID":"ee7b1eaf-3df1-49ba-b8af-522596a7e862","Type":"ContainerStarted","Data":"fd0f08ecf27dd60c5098664e6dcd770ea8b596ec3960be191614b541cfb0cc56"}
Jul 29 00:43:16.153921 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:16.153904    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw" event={"ID":"ee7b1eaf-3df1-49ba-b8af-522596a7e862","Type":"ContainerStarted","Data":"e1495230515253ad7002a26479e9393cf6d37e0437734a00cd4973d8a2bcc326"}
Jul 29 00:43:16.154426 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:16.153929    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"
Jul 29 00:43:16.156374 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:16.156344    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4wkq9\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4wkq9"
Jul 29 00:43:16.158167 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:16.158141    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-svlhk\" is forbidden: User \"system:node:ip-10-0-132-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-227.ec2.internal' and this object" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-svlhk"
Jul 29 00:43:16.180932 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:16.180860    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw" podStartSLOduration=2.180842905 podStartE2EDuration="2.180842905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:43:16.177784857 +0000 UTC m=+426.486712705" watchObservedRunningTime="2026-07-29 00:43:16.180842905 +0000 UTC m=+426.489770870"
Jul 29 00:43:16.222189 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:16.222153    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e75683c-cc23-4d4b-b224-97dd7f1b6107" path="/var/lib/kubelet/pods/6e75683c-cc23-4d4b-b224-97dd7f1b6107/volumes"
Jul 29 00:43:16.222539 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:16.222525    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9a01a9-0c60-41ff-8824-8cfba76a8160" path="/var/lib/kubelet/pods/eb9a01a9-0c60-41ff-8824-8cfba76a8160/volumes"
Jul 29 00:43:17.261507 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:17.261470    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:17.391944 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:17.391911    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:20.055619 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:20.055582    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:20.159745 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:20.159713    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:23.129578 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.129539    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d9fbffb86-gwb6j" podUID="0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" containerName="console" containerID="cri-o://82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25" gracePeriod=15
Jul 29 00:43:23.384234 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.384148    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:43:23.505315 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505278    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-oauth-serving-cert\") pod \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") "
Jul 29 00:43:23.505514 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505330    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-oauth-config\") pod \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") "
Jul 29 00:43:23.505514 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505349    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-service-ca\") pod \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") "
Jul 29 00:43:23.505514 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505384    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-trusted-ca-bundle\") pod \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") "
Jul 29 00:43:23.505514 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505418    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfjxc\" (UniqueName: \"kubernetes.io/projected/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-kube-api-access-pfjxc\") pod \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") "
Jul 29 00:43:23.505514 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505437    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-serving-cert\") pod \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") "
Jul 29 00:43:23.505514 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505497    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-config\") pod \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\" (UID: \"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda\") "
Jul 29 00:43:23.505842 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505750    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-service-ca" (OuterVolumeSpecName: "service-ca") pod "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" (UID: "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:43:23.505904 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505779    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" (UID: "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:43:23.505949 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505886    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" (UID: "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:43:23.505996 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.505969    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-config" (OuterVolumeSpecName: "console-config") pod "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" (UID: "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:43:23.507731 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.507696    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-kube-api-access-pfjxc" (OuterVolumeSpecName: "kube-api-access-pfjxc") pod "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" (UID: "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda"). InnerVolumeSpecName "kube-api-access-pfjxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:43:23.507877 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.507750    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" (UID: "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:43:23.507877 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.507765    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" (UID: "0a5d7591-95dc-40c5-a9d0-da8acc5d8bda"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:43:23.594408 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.594373    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:23.606588 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.606552    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:23.606588 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.606585    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-oauth-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:23.606588 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.606595    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-oauth-config\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:23.606831 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.606603    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:23.606831 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.606612    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-trusted-ca-bundle\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:23.606831 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.606621    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfjxc\" (UniqueName: \"kubernetes.io/projected/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-kube-api-access-pfjxc\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:23.606831 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.606631    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda-console-serving-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:43:23.706604 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:23.706513    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:24.198357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:24.198317    2586 generic.go:356] "Generic (PLEG): container finished" podID="0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" containerID="82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25" exitCode=0
Jul 29 00:43:24.198357 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:24.198361    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9fbffb86-gwb6j"
Jul 29 00:43:24.198896 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:24.198398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9fbffb86-gwb6j" event={"ID":"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda","Type":"ContainerDied","Data":"82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25"}
Jul 29 00:43:24.198896 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:24.198452    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9fbffb86-gwb6j" event={"ID":"0a5d7591-95dc-40c5-a9d0-da8acc5d8bda","Type":"ContainerDied","Data":"d2ce6ee0f20119488b7ce03dff106160593a33cd24fe3231569cfd27ffdf1b59"}
Jul 29 00:43:24.198896 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:24.198474    2586 scope.go:117] "RemoveContainer" containerID="82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25"
Jul 29 00:43:24.207757 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:24.207740    2586 scope.go:117] "RemoveContainer" containerID="82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25"
Jul 29 00:43:24.208013 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:43:24.207994    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25\": container with ID starting with 82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25 not found: ID does not exist" containerID="82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25"
Jul 29 00:43:24.208060 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:24.208022    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25"} err="failed to get container status \"82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25\": rpc error: code = NotFound desc = could not find container \"82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25\": container with ID starting with 82b1b1916f453e64f7c8c08a0beb717cde36c6cdc876d68f13b9ea5b104acb25 not found: ID does not exist"
Jul 29 00:43:24.223017 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:24.222986    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9fbffb86-gwb6j"]
Jul 29 00:43:24.227264 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:24.227238    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d9fbffb86-gwb6j"]
Jul 29 00:43:26.221087 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:26.221049    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" path="/var/lib/kubelet/pods/0a5d7591-95dc-40c5-a9d0-da8acc5d8bda/volumes"
Jul 29 00:43:27.164492 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:27.164452    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lm7xw"
Jul 29 00:43:32.419464 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:32.419375    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:32.522927 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:32.522886    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:35.037106 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:35.037069    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:35.137360 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:35.137323    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:38.227749 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:38.227718    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:38.328844 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:38.328814    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:41.568729 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.568684    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"]
Jul 29 00:43:41.569131 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.569053    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" containerName="console"
Jul 29 00:43:41.569131 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.569067    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" containerName="console"
Jul 29 00:43:41.569244 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.569134    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a5d7591-95dc-40c5-a9d0-da8acc5d8bda" containerName="console"
Jul 29 00:43:41.571065 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.571046    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:41.573786 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.573768    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sq5mr\""
Jul 29 00:43:41.585716 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.585688    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"]
Jul 29 00:43:41.652888 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.652857    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:41.654987 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.654964    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lpvm5\" (UID: \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:41.655099 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.655042    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fzj\" (UniqueName: \"kubernetes.io/projected/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-kube-api-access-m9fzj\") pod \"kuadrant-operator-controller-manager-86878c769f-lpvm5\" (UID: \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:41.755718 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.755677    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fzj\" (UniqueName: \"kubernetes.io/projected/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-kube-api-access-m9fzj\") pod \"kuadrant-operator-controller-manager-86878c769f-lpvm5\" (UID: \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:41.755718 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.755722    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lpvm5\" (UID: \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:41.756100 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.756080    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lpvm5\" (UID: \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:41.762962 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.762932    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:41.786848 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.786816    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fzj\" (UniqueName: \"kubernetes.io/projected/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-kube-api-access-m9fzj\") pod \"kuadrant-operator-controller-manager-86878c769f-lpvm5\" (UID: \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:41.881758 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:41.881658    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:42.018977 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:42.018952    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"]
Jul 29 00:43:42.021959 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:43:42.021929    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6af2a8_9cbd_45e7_be67_a1a756e18d09.slice/crio-de5eeb5e89b6ec30e870d4bbd37213b2797d2ff4214adececfa0562fc343b684 WatchSource:0}: Error finding container de5eeb5e89b6ec30e870d4bbd37213b2797d2ff4214adececfa0562fc343b684: Status 404 returned error can't find the container with id de5eeb5e89b6ec30e870d4bbd37213b2797d2ff4214adececfa0562fc343b684
Jul 29 00:43:42.296321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:42.296223    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5" event={"ID":"0c6af2a8-9cbd-45e7-be67-a1a756e18d09","Type":"ContainerStarted","Data":"19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92"}
Jul 29 00:43:42.296321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:42.296289    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:42.296321 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:42.296309    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5" event={"ID":"0c6af2a8-9cbd-45e7-be67-a1a756e18d09","Type":"ContainerStarted","Data":"de5eeb5e89b6ec30e870d4bbd37213b2797d2ff4214adececfa0562fc343b684"}
Jul 29 00:43:42.329404 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:42.329355    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5" podStartSLOduration=1.329340647 podStartE2EDuration="1.329340647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:43:42.326327718 +0000 UTC m=+452.635255559" watchObservedRunningTime="2026-07-29 00:43:42.329340647 +0000 UTC m=+452.638268490"
Jul 29 00:43:46.588640 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:46.588608    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:46.688846 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:46.688813    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:50.272294 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:50.272256    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:50.369993 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:50.369954    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:53.304662 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:53.304626    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:43:57.651553 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:57.651515    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:43:57.812318 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:43:57.812280    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:02.024431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:02.024388    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:02.130715 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:02.130685    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:02.231744 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:02.231714    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:02.859330 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:02.859295    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:02.919607 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:02.919572    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:02.985580 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:02.985543    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-dv8hf"]
Jul 29 00:44:02.989855 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:02.989826    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dv8hf"
Jul 29 00:44:02.992681 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:02.992657    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h64br\""
Jul 29 00:44:02.996443 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:02.996414    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dv8hf"]
Jul 29 00:44:03.156807 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:03.156720    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g42lj\" (UniqueName: \"kubernetes.io/projected/2cc011e1-a995-462e-b6b8-7bc6ed27d670-kube-api-access-g42lj\") pod \"authorino-75d556bd77-dv8hf\" (UID: \"2cc011e1-a995-462e-b6b8-7bc6ed27d670\") " pod="kuadrant-system/authorino-75d556bd77-dv8hf"
Jul 29 00:44:03.257549 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:03.257512    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g42lj\" (UniqueName: \"kubernetes.io/projected/2cc011e1-a995-462e-b6b8-7bc6ed27d670-kube-api-access-g42lj\") pod \"authorino-75d556bd77-dv8hf\" (UID: \"2cc011e1-a995-462e-b6b8-7bc6ed27d670\") " pod="kuadrant-system/authorino-75d556bd77-dv8hf"
Jul 29 00:44:03.265961 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:03.265937    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g42lj\" (UniqueName: \"kubernetes.io/projected/2cc011e1-a995-462e-b6b8-7bc6ed27d670-kube-api-access-g42lj\") pod \"authorino-75d556bd77-dv8hf\" (UID: \"2cc011e1-a995-462e-b6b8-7bc6ed27d670\") " pod="kuadrant-system/authorino-75d556bd77-dv8hf"
Jul 29 00:44:03.300684 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:03.300650    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dv8hf"
Jul 29 00:44:03.436114 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:03.436089    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dv8hf"]
Jul 29 00:44:03.438015 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:44:03.437989    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cc011e1_a995_462e_b6b8_7bc6ed27d670.slice/crio-e3d3b87333e834ee32b34e04e003bff8911f09827456248f8f0b1fd155acad57 WatchSource:0}: Error finding container e3d3b87333e834ee32b34e04e003bff8911f09827456248f8f0b1fd155acad57: Status 404 returned error can't find the container with id e3d3b87333e834ee32b34e04e003bff8911f09827456248f8f0b1fd155acad57
Jul 29 00:44:04.419000 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:04.418962    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dv8hf" event={"ID":"2cc011e1-a995-462e-b6b8-7bc6ed27d670","Type":"ContainerStarted","Data":"e3d3b87333e834ee32b34e04e003bff8911f09827456248f8f0b1fd155acad57"}
Jul 29 00:44:07.438878 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:07.438840    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dv8hf" event={"ID":"2cc011e1-a995-462e-b6b8-7bc6ed27d670","Type":"ContainerStarted","Data":"ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7"}
Jul 29 00:44:07.456597 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:07.456533    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-dv8hf" podStartSLOduration=2.324423257 podStartE2EDuration="5.456514729s" totalImagesPullingTime="3.132091472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:44:07.454273477 +0000 UTC m=+477.763201313" watchObservedRunningTime="2026-07-29 00:44:07.456514729 +0000 UTC m=+477.765442570"
Jul 29 00:44:20.459302 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:20.459273    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:20.560431 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:20.560392    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:23.259802 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:23.259730    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:23.357847 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:23.357816    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:23.719900 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:23.719865    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:35.527105 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:35.527073    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:35.742743 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:35.742709    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:36.066798 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.066760    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d5bbd879f-59spd"]
Jul 29 00:44:36.070278 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.070261    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:44:36.073338 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.073319    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 00:44:36.077501 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.077472    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d5bbd879f-59spd"]
Jul 29 00:44:36.118572 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.118533    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp4m8\" (UniqueName: \"kubernetes.io/projected/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-kube-api-access-hp4m8\") pod \"authorino-6d5bbd879f-59spd\" (UID: \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\") " pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:44:36.118750 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.118594    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-tls-cert\") pod \"authorino-6d5bbd879f-59spd\" (UID: \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\") " pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:44:36.219625 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.219584    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp4m8\" (UniqueName: \"kubernetes.io/projected/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-kube-api-access-hp4m8\") pod \"authorino-6d5bbd879f-59spd\" (UID: \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\") " pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:44:36.219819 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.219667    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-tls-cert\") pod \"authorino-6d5bbd879f-59spd\" (UID: \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\") " pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:44:36.222260 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.222232    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-tls-cert\") pod \"authorino-6d5bbd879f-59spd\" (UID: \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\") " pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:44:36.227764 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.227742    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp4m8\" (UniqueName: \"kubernetes.io/projected/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-kube-api-access-hp4m8\") pod \"authorino-6d5bbd879f-59spd\" (UID: \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\") " pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:44:36.381962 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.381876    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:44:36.512568 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.512534    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d5bbd879f-59spd"]
Jul 29 00:44:36.514182 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:44:36.514158    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d82b750_a0f6_4a3d_b8b8_00e791d1275a.slice/crio-6f81c77c4185a3d4cbba94a47a790c59da289716fbcd571d00225b46c0095102 WatchSource:0}: Error finding container 6f81c77c4185a3d4cbba94a47a790c59da289716fbcd571d00225b46c0095102: Status 404 returned error can't find the container with id 6f81c77c4185a3d4cbba94a47a790c59da289716fbcd571d00225b46c0095102
Jul 29 00:44:36.587708 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:36.587663    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d5bbd879f-59spd" event={"ID":"0d82b750-a0f6-4a3d-b8b8-00e791d1275a","Type":"ContainerStarted","Data":"6f81c77c4185a3d4cbba94a47a790c59da289716fbcd571d00225b46c0095102"}
Jul 29 00:44:37.594134 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:37.594099    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d5bbd879f-59spd" event={"ID":"0d82b750-a0f6-4a3d-b8b8-00e791d1275a","Type":"ContainerStarted","Data":"3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35"}
Jul 29 00:44:37.613719 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:37.613664    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d5bbd879f-59spd" podStartSLOduration=1.2918253069999999 podStartE2EDuration="1.61364646s" totalImagesPullingTime="321.821153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:44:37.612883865 +0000 UTC m=+507.921811711" watchObservedRunningTime="2026-07-29 00:44:37.61364646 +0000 UTC m=+507.922574301"
Jul 29 00:44:37.650828 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:37.650792    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dv8hf"]
Jul 29 00:44:37.651021 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:37.650997    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-dv8hf" podUID="2cc011e1-a995-462e-b6b8-7bc6ed27d670" containerName="authorino" containerID="cri-o://ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7" gracePeriod=30
Jul 29 00:44:37.895749 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:37.895723    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dv8hf"
Jul 29 00:44:37.938608 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:37.938573    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g42lj\" (UniqueName: \"kubernetes.io/projected/2cc011e1-a995-462e-b6b8-7bc6ed27d670-kube-api-access-g42lj\") pod \"2cc011e1-a995-462e-b6b8-7bc6ed27d670\" (UID: \"2cc011e1-a995-462e-b6b8-7bc6ed27d670\") "
Jul 29 00:44:37.940627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:37.940591    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc011e1-a995-462e-b6b8-7bc6ed27d670-kube-api-access-g42lj" (OuterVolumeSpecName: "kube-api-access-g42lj") pod "2cc011e1-a995-462e-b6b8-7bc6ed27d670" (UID: "2cc011e1-a995-462e-b6b8-7bc6ed27d670"). InnerVolumeSpecName "kube-api-access-g42lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:44:38.039806 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.039759    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g42lj\" (UniqueName: \"kubernetes.io/projected/2cc011e1-a995-462e-b6b8-7bc6ed27d670-kube-api-access-g42lj\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:44:38.600870 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.600835    2586 generic.go:356] "Generic (PLEG): container finished" podID="2cc011e1-a995-462e-b6b8-7bc6ed27d670" containerID="ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7" exitCode=0
Jul 29 00:44:38.601362 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.600872    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dv8hf" event={"ID":"2cc011e1-a995-462e-b6b8-7bc6ed27d670","Type":"ContainerDied","Data":"ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7"}
Jul 29 00:44:38.601362 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.600914    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dv8hf" event={"ID":"2cc011e1-a995-462e-b6b8-7bc6ed27d670","Type":"ContainerDied","Data":"e3d3b87333e834ee32b34e04e003bff8911f09827456248f8f0b1fd155acad57"}
Jul 29 00:44:38.601362 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.600932    2586 scope.go:117] "RemoveContainer" containerID="ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7"
Jul 29 00:44:38.601362 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.600885    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dv8hf"
Jul 29 00:44:38.609568 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.609550    2586 scope.go:117] "RemoveContainer" containerID="ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7"
Jul 29 00:44:38.609850 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:44:38.609830    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7\": container with ID starting with ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7 not found: ID does not exist" containerID="ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7"
Jul 29 00:44:38.609913 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.609860    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7"} err="failed to get container status \"ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7\": rpc error: code = NotFound desc = could not find container \"ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7\": container with ID starting with ed13839dcc15bf77b19d3bd0eee75d25ff37013378df134a46deba6dd5e7b3b7 not found: ID does not exist"
Jul 29 00:44:38.616949 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.616924    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dv8hf"]
Jul 29 00:44:38.620641 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:38.620614    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dv8hf"]
Jul 29 00:44:40.221990 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:40.221950    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cc011e1-a995-462e-b6b8-7bc6ed27d670" path="/var/lib/kubelet/pods/2cc011e1-a995-462e-b6b8-7bc6ed27d670/volumes"
Jul 29 00:44:42.156059 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.156021    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bb75f66b6-lhx8v"]
Jul 29 00:44:42.156573 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.156547    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc011e1-a995-462e-b6b8-7bc6ed27d670" containerName="authorino"
Jul 29 00:44:42.156653 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.156567    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc011e1-a995-462e-b6b8-7bc6ed27d670" containerName="authorino"
Jul 29 00:44:42.156754 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.156731    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc011e1-a995-462e-b6b8-7bc6ed27d670" containerName="authorino"
Jul 29 00:44:42.160649 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.160625    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:42.161578 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.161539    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:42.163716 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.163662    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 00:44:42.163917 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.163900    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 00:44:42.164004 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.163948    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xnhp8\""
Jul 29 00:44:42.171379 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.171343    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bb75f66b6-lhx8v"]
Jul 29 00:44:42.226377 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.226344    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:42.274901 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.274867    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59zb\" (UniqueName: \"kubernetes.io/projected/49b66030-0218-46ad-bf8e-3a1f51aed88d-kube-api-access-j59zb\") pod \"maas-controller-5bb75f66b6-lhx8v\" (UID: \"49b66030-0218-46ad-bf8e-3a1f51aed88d\") " pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:42.275050 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.274937    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49b66030-0218-46ad-bf8e-3a1f51aed88d-webhook-cert\") pod \"maas-controller-5bb75f66b6-lhx8v\" (UID: \"49b66030-0218-46ad-bf8e-3a1f51aed88d\") " pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:42.317931 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.317895    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:42.376012 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.375976    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59zb\" (UniqueName: \"kubernetes.io/projected/49b66030-0218-46ad-bf8e-3a1f51aed88d-kube-api-access-j59zb\") pod \"maas-controller-5bb75f66b6-lhx8v\" (UID: \"49b66030-0218-46ad-bf8e-3a1f51aed88d\") " pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:42.376172 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.376054    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49b66030-0218-46ad-bf8e-3a1f51aed88d-webhook-cert\") pod \"maas-controller-5bb75f66b6-lhx8v\" (UID: \"49b66030-0218-46ad-bf8e-3a1f51aed88d\") " pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:42.378420 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.378401    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49b66030-0218-46ad-bf8e-3a1f51aed88d-webhook-cert\") pod \"maas-controller-5bb75f66b6-lhx8v\" (UID: \"49b66030-0218-46ad-bf8e-3a1f51aed88d\") " pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:42.393470 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.393439    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59zb\" (UniqueName: \"kubernetes.io/projected/49b66030-0218-46ad-bf8e-3a1f51aed88d-kube-api-access-j59zb\") pod \"maas-controller-5bb75f66b6-lhx8v\" (UID: \"49b66030-0218-46ad-bf8e-3a1f51aed88d\") " pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:42.413041 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.412970    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:42.472934 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.472908    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:42.605912 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.605888    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bb75f66b6-lhx8v"]
Jul 29 00:44:42.607862 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:44:42.607835    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b66030_0218_46ad_bf8e_3a1f51aed88d.slice/crio-791443b40fed5b3e82bdd2ff2ea8eb97eb1a34fad80338cb0937af8750db181f WatchSource:0}: Error finding container 791443b40fed5b3e82bdd2ff2ea8eb97eb1a34fad80338cb0937af8750db181f: Status 404 returned error can't find the container with id 791443b40fed5b3e82bdd2ff2ea8eb97eb1a34fad80338cb0937af8750db181f
Jul 29 00:44:42.625385 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:42.625357    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v" event={"ID":"49b66030-0218-46ad-bf8e-3a1f51aed88d","Type":"ContainerStarted","Data":"791443b40fed5b3e82bdd2ff2ea8eb97eb1a34fad80338cb0937af8750db181f"}
Jul 29 00:44:45.184511 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:45.184478    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:45.336024 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:45.335993    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:44:45.644891 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:45.644790    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v" event={"ID":"49b66030-0218-46ad-bf8e-3a1f51aed88d","Type":"ContainerStarted","Data":"c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa"}
Jul 29 00:44:45.644891 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:45.644861    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:45.670443 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:45.670389    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v" podStartSLOduration=1.245638128 podStartE2EDuration="3.670370558s" totalImagesPullingTime="2.42473243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:44:45.66844712 +0000 UTC m=+515.977374955" watchObservedRunningTime="2026-07-29 00:44:45.670370558 +0000 UTC m=+515.979298401"
Jul 29 00:44:56.287915 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.287858    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bb75f66b6-lhx8v"]
Jul 29 00:44:56.288664 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.288206    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v" podUID="49b66030-0218-46ad-bf8e-3a1f51aed88d" containerName="manager" containerID="cri-o://c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa" gracePeriod=30
Jul 29 00:44:56.289795 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.289766    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:56.535819 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.535789    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:56.605797 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.605763    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j59zb\" (UniqueName: \"kubernetes.io/projected/49b66030-0218-46ad-bf8e-3a1f51aed88d-kube-api-access-j59zb\") pod \"49b66030-0218-46ad-bf8e-3a1f51aed88d\" (UID: \"49b66030-0218-46ad-bf8e-3a1f51aed88d\") "
Jul 29 00:44:56.605964 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.605887    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49b66030-0218-46ad-bf8e-3a1f51aed88d-webhook-cert\") pod \"49b66030-0218-46ad-bf8e-3a1f51aed88d\" (UID: \"49b66030-0218-46ad-bf8e-3a1f51aed88d\") "
Jul 29 00:44:56.607968 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.607936    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b66030-0218-46ad-bf8e-3a1f51aed88d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "49b66030-0218-46ad-bf8e-3a1f51aed88d" (UID: "49b66030-0218-46ad-bf8e-3a1f51aed88d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:44:56.608077 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.607989    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b66030-0218-46ad-bf8e-3a1f51aed88d-kube-api-access-j59zb" (OuterVolumeSpecName: "kube-api-access-j59zb") pod "49b66030-0218-46ad-bf8e-3a1f51aed88d" (UID: "49b66030-0218-46ad-bf8e-3a1f51aed88d"). InnerVolumeSpecName "kube-api-access-j59zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:44:56.701367 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.701332    2586 generic.go:356] "Generic (PLEG): container finished" podID="49b66030-0218-46ad-bf8e-3a1f51aed88d" containerID="c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa" exitCode=0
Jul 29 00:44:56.701549 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.701375    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v"
Jul 29 00:44:56.701549 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.701421    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v" event={"ID":"49b66030-0218-46ad-bf8e-3a1f51aed88d","Type":"ContainerDied","Data":"c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa"}
Jul 29 00:44:56.701549 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.701472    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb75f66b6-lhx8v" event={"ID":"49b66030-0218-46ad-bf8e-3a1f51aed88d","Type":"ContainerDied","Data":"791443b40fed5b3e82bdd2ff2ea8eb97eb1a34fad80338cb0937af8750db181f"}
Jul 29 00:44:56.701549 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.701494    2586 scope.go:117] "RemoveContainer" containerID="c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa"
Jul 29 00:44:56.706490 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.706467    2586 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49b66030-0218-46ad-bf8e-3a1f51aed88d-webhook-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:44:56.706490 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.706490    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59zb\" (UniqueName: \"kubernetes.io/projected/49b66030-0218-46ad-bf8e-3a1f51aed88d-kube-api-access-j59zb\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:44:56.710159 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.710143    2586 scope.go:117] "RemoveContainer" containerID="c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa"
Jul 29 00:44:56.710430 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:44:56.710411    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa\": container with ID starting with c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa not found: ID does not exist" containerID="c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa"
Jul 29 00:44:56.710478 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.710438    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa"} err="failed to get container status \"c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa\": rpc error: code = NotFound desc = could not find container \"c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa\": container with ID starting with c22899b13b112923dae25ffeee11eb7dc97f865daf6d82ce4d6fb34001ce01aa not found: ID does not exist"
Jul 29 00:44:56.722991 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.722971    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bb75f66b6-lhx8v"]
Jul 29 00:44:56.727012 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:56.726993    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bb75f66b6-lhx8v"]
Jul 29 00:44:58.221935 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:44:58.221903    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b66030-0218-46ad-bf8e-3a1f51aed88d" path="/var/lib/kubelet/pods/49b66030-0218-46ad-bf8e-3a1f51aed88d/volumes"
Jul 29 00:45:08.320129 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:08.320088    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:17.482915 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:17.482870    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:17.589139 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:17.589102    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:18.268307 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.268272    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:18.450910 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.450875    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:18.757841 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.757802    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:18.861249 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.861190    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:18.896999 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.896960    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5cvtz"]
Jul 29 00:45:18.898114 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.898084    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b66030-0218-46ad-bf8e-3a1f51aed88d" containerName="manager"
Jul 29 00:45:18.898114 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.898109    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b66030-0218-46ad-bf8e-3a1f51aed88d" containerName="manager"
Jul 29 00:45:18.898316 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.898281    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b66030-0218-46ad-bf8e-3a1f51aed88d" containerName="manager"
Jul 29 00:45:18.906954 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.906926    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:18.908876 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.908842    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5cvtz"]
Jul 29 00:45:18.909670 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.909648    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 29 00:45:18.909783 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:18.909736    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6bw7v\""
Jul 29 00:45:19.101317 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:19.101277    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a369ca0-d428-4bee-bc25-23a4ea6dd7fe-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5cvtz\" (UID: \"5a369ca0-d428-4bee-bc25-23a4ea6dd7fe\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:19.101502 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:19.101411    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s22f7\" (UniqueName: \"kubernetes.io/projected/5a369ca0-d428-4bee-bc25-23a4ea6dd7fe-kube-api-access-s22f7\") pod \"payload-processing-5cc6cd444b-5cvtz\" (UID: \"5a369ca0-d428-4bee-bc25-23a4ea6dd7fe\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:19.202563 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:19.202521    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a369ca0-d428-4bee-bc25-23a4ea6dd7fe-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5cvtz\" (UID: \"5a369ca0-d428-4bee-bc25-23a4ea6dd7fe\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:19.202769 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:19.202599    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s22f7\" (UniqueName: \"kubernetes.io/projected/5a369ca0-d428-4bee-bc25-23a4ea6dd7fe-kube-api-access-s22f7\") pod \"payload-processing-5cc6cd444b-5cvtz\" (UID: \"5a369ca0-d428-4bee-bc25-23a4ea6dd7fe\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:19.203451 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:19.203425    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a369ca0-d428-4bee-bc25-23a4ea6dd7fe-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5cvtz\" (UID: \"5a369ca0-d428-4bee-bc25-23a4ea6dd7fe\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:19.210318 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:19.210294    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s22f7\" (UniqueName: \"kubernetes.io/projected/5a369ca0-d428-4bee-bc25-23a4ea6dd7fe-kube-api-access-s22f7\") pod \"payload-processing-5cc6cd444b-5cvtz\" (UID: \"5a369ca0-d428-4bee-bc25-23a4ea6dd7fe\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:19.219209 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:19.219172    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:19.356546 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:19.356520    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5cvtz"]
Jul 29 00:45:19.357994 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:45:19.357969    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a369ca0_d428_4bee_bc25_23a4ea6dd7fe.slice/crio-4085362f3b6b1e05bdc79033cbec348fab50fd3800013768c1f84f262ecd1f51 WatchSource:0}: Error finding container 4085362f3b6b1e05bdc79033cbec348fab50fd3800013768c1f84f262ecd1f51: Status 404 returned error can't find the container with id 4085362f3b6b1e05bdc79033cbec348fab50fd3800013768c1f84f262ecd1f51
Jul 29 00:45:19.830914 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:19.830874    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz" event={"ID":"5a369ca0-d428-4bee-bc25-23a4ea6dd7fe","Type":"ContainerStarted","Data":"4085362f3b6b1e05bdc79033cbec348fab50fd3800013768c1f84f262ecd1f51"}
Jul 29 00:45:20.837911 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:20.837877    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz" event={"ID":"5a369ca0-d428-4bee-bc25-23a4ea6dd7fe","Type":"ContainerStarted","Data":"4c1cf370cb672053afafc9da5ef46be9168cbad0072a6a1cb0a222b2470a470d"}
Jul 29 00:45:20.838287 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:20.837937    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:20.856966 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:20.856913    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz" podStartSLOduration=1.499931939 podStartE2EDuration="2.856897766s" totalImagesPullingTime="1.356965827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:45:20.855408147 +0000 UTC m=+551.164336002" watchObservedRunningTime="2026-07-29 00:45:20.856897766 +0000 UTC m=+551.165825608"
Jul 29 00:45:29.417383 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:29.417345    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:29.512684 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:29.512654    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:31.844916 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:31.844883    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5cvtz"
Jul 29 00:45:54.619067 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.619027    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"]
Jul 29 00:45:54.622610 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.622592    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.625526 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.625503    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 00:45:54.625620 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.625585    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 00:45:54.626799 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.626781    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7rgtn\""
Jul 29 00:45:54.626924 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.626785    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 29 00:45:54.636060 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.636018    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"]
Jul 29 00:45:54.711639 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.711605    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:45:54.713944 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.713919    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.714052 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.713950    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.714052 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.713983    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.714125 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.714081    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.714169 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.714124    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04626e14-481a-414c-97a0-6e0d3de7a19e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.714169 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.714141    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhgh\" (UniqueName: \"kubernetes.io/projected/04626e14-481a-414c-97a0-6e0d3de7a19e-kube-api-access-qlhgh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.815328 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.815282    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.815328 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.815329    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.815584 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.815374    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.815584 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.815406    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.815584 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.815462    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04626e14-481a-414c-97a0-6e0d3de7a19e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.815584 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.815486    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhgh\" (UniqueName: \"kubernetes.io/projected/04626e14-481a-414c-97a0-6e0d3de7a19e-kube-api-access-qlhgh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.815844 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.815813    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.816004 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.815977    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.816004 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.815997    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.817728 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.817697    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04626e14-481a-414c-97a0-6e0d3de7a19e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.817971 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.817955    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04626e14-481a-414c-97a0-6e0d3de7a19e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.823993 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.823967    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhgh\" (UniqueName: \"kubernetes.io/projected/04626e14-481a-414c-97a0-6e0d3de7a19e-kube-api-access-qlhgh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm\" (UID: \"04626e14-481a-414c-97a0-6e0d3de7a19e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:54.934164 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:54.934075    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:45:55.078682 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:55.078654    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"]
Jul 29 00:45:55.080513 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:45:55.080485    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04626e14_481a_414c_97a0_6e0d3de7a19e.slice/crio-450aa0ef6c9283064bdba8da378426b7bcdb46c05c39849e5dc20c8764662d1e WatchSource:0}: Error finding container 450aa0ef6c9283064bdba8da378426b7bcdb46c05c39849e5dc20c8764662d1e: Status 404 returned error can't find the container with id 450aa0ef6c9283064bdba8da378426b7bcdb46c05c39849e5dc20c8764662d1e
Jul 29 00:45:56.029109 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:56.029068    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm" event={"ID":"04626e14-481a-414c-97a0-6e0d3de7a19e","Type":"ContainerStarted","Data":"450aa0ef6c9283064bdba8da378426b7bcdb46c05c39849e5dc20c8764662d1e"}
Jul 29 00:45:57.037813 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:57.037776    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm" event={"ID":"04626e14-481a-414c-97a0-6e0d3de7a19e","Type":"ContainerStarted","Data":"847f9a8230aa22bf1819c3ef6387c59964a9853c583ad0ef4ee4dd5df407408a"}
Jul 29 00:45:57.059264 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:45:57.059183    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm" podStartSLOduration=1.901702665 podStartE2EDuration="3.059165305s" totalImagesPullingTime="1.15746264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:45:57.057632214 +0000 UTC m=+587.366560057" watchObservedRunningTime="2026-07-29 00:45:57.059165305 +0000 UTC m=+587.368093147"
Jul 29 00:46:04.934688 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:04.934647    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:46:04.935156 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:04.934726    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:46:04.947562 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:04.947527    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:46:05.092102 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:05.092070    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm"
Jul 29 00:46:10.928307 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:10.928270    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:46:23.003486 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:23.003448    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:46:25.225621 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.225586    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"]
Jul 29 00:46:25.229162 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.229141    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.232277 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.232249    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 29 00:46:25.238781 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.238746    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"]
Jul 29 00:46:25.247849 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.247815    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:46:25.270415 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.270374    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.270415 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.270418    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.270676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.270521    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.270676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.270573    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k6hh\" (UniqueName: \"kubernetes.io/projected/722e2b17-ec12-4396-8303-23c324aaff37-kube-api-access-4k6hh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.270676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.270612    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.270812 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.270687    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722e2b17-ec12-4396-8303-23c324aaff37-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.371492 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.371452    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.371702 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.371509    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.371702 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.371628    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6hh\" (UniqueName: \"kubernetes.io/projected/722e2b17-ec12-4396-8303-23c324aaff37-kube-api-access-4k6hh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.371702 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.371663    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.371885 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.371726    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722e2b17-ec12-4396-8303-23c324aaff37-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.371885 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.371764    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.372023 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.371888    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.372150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.372119    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.372150 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.372140    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.373891 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.373871    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722e2b17-ec12-4396-8303-23c324aaff37-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.374422 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.374407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722e2b17-ec12-4396-8303-23c324aaff37-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.379666 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.379643    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6hh\" (UniqueName: \"kubernetes.io/projected/722e2b17-ec12-4396-8303-23c324aaff37-kube-api-access-4k6hh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46k88\" (UID: \"722e2b17-ec12-4396-8303-23c324aaff37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.541312 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.541181    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:25.691918 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.691885    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"]
Jul 29 00:46:25.693164 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:46:25.693133    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722e2b17_ec12_4396_8303_23c324aaff37.slice/crio-ef83d2f59607c7a2d4e0a7a8fd94901ca52a1cd778bd0b296adf496646e9f801 WatchSource:0}: Error finding container ef83d2f59607c7a2d4e0a7a8fd94901ca52a1cd778bd0b296adf496646e9f801: Status 404 returned error can't find the container with id ef83d2f59607c7a2d4e0a7a8fd94901ca52a1cd778bd0b296adf496646e9f801
Jul 29 00:46:25.694548 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:25.694526    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 00:46:26.191977 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:26.191878    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88" event={"ID":"722e2b17-ec12-4396-8303-23c324aaff37","Type":"ContainerStarted","Data":"1f093e4b50d0cc7c4177f465c25f7004b0104090dc06f69d84bcf7c9e0aed5fd"}
Jul 29 00:46:26.191977 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:26.191945    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88" event={"ID":"722e2b17-ec12-4396-8303-23c324aaff37","Type":"ContainerStarted","Data":"ef83d2f59607c7a2d4e0a7a8fd94901ca52a1cd778bd0b296adf496646e9f801"}
Jul 29 00:46:26.212433 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:26.212374    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88" podStartSLOduration=1.027655983 podStartE2EDuration="1.212356724s" totalImagesPullingTime="184.700741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:46:26.210948965 +0000 UTC m=+616.519876808" watchObservedRunningTime="2026-07-29 00:46:26.212356724 +0000 UTC m=+616.521284544"
Jul 29 00:46:35.541569 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:35.541463    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:35.542018 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:35.541764    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:35.554235 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:35.554185    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:36.262435 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:36.262405    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46k88"
Jul 29 00:46:45.162423 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:45.162387    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:46:45.262825 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:45.262790    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:46:50.793735 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:46:50.793691    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:47:00.843279 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:47:00.843242    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:47:12.052452 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:47:12.052419    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:47:51.354864 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:47:51.354831    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:48:15.979320 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:15.979283    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:48:16.147910 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:16.147870    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:48:17.165717 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:17.165678    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:48:17.263627 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:17.263583    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:48:17.682353 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:17.681781    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d5bbd879f-59spd"]
Jul 29 00:48:17.682353 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:17.682009    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d5bbd879f-59spd" podUID="0d82b750-a0f6-4a3d-b8b8-00e791d1275a" containerName="authorino" containerID="cri-o://3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35" gracePeriod=30
Jul 29 00:48:17.930027 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:17.930001    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:48:17.982349 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:17.982261    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp4m8\" (UniqueName: \"kubernetes.io/projected/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-kube-api-access-hp4m8\") pod \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\" (UID: \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\") "
Jul 29 00:48:17.982529 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:17.982359    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-tls-cert\") pod \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\" (UID: \"0d82b750-a0f6-4a3d-b8b8-00e791d1275a\") "
Jul 29 00:48:17.984594 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:17.984553    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-kube-api-access-hp4m8" (OuterVolumeSpecName: "kube-api-access-hp4m8") pod "0d82b750-a0f6-4a3d-b8b8-00e791d1275a" (UID: "0d82b750-a0f6-4a3d-b8b8-00e791d1275a"). InnerVolumeSpecName "kube-api-access-hp4m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:48:17.993301 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:17.993253    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0d82b750-a0f6-4a3d-b8b8-00e791d1275a" (UID: "0d82b750-a0f6-4a3d-b8b8-00e791d1275a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:48:18.083275 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.083226    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp4m8\" (UniqueName: \"kubernetes.io/projected/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-kube-api-access-hp4m8\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:48:18.083275 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.083266    2586 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d82b750-a0f6-4a3d-b8b8-00e791d1275a-tls-cert\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:48:18.803556 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.803520    2586 generic.go:356] "Generic (PLEG): container finished" podID="0d82b750-a0f6-4a3d-b8b8-00e791d1275a" containerID="3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35" exitCode=0
Jul 29 00:48:18.803969 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.803562    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d5bbd879f-59spd"
Jul 29 00:48:18.803969 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.803609    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d5bbd879f-59spd" event={"ID":"0d82b750-a0f6-4a3d-b8b8-00e791d1275a","Type":"ContainerDied","Data":"3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35"}
Jul 29 00:48:18.803969 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.803648    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d5bbd879f-59spd" event={"ID":"0d82b750-a0f6-4a3d-b8b8-00e791d1275a","Type":"ContainerDied","Data":"6f81c77c4185a3d4cbba94a47a790c59da289716fbcd571d00225b46c0095102"}
Jul 29 00:48:18.803969 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.803664    2586 scope.go:117] "RemoveContainer" containerID="3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35"
Jul 29 00:48:18.813493 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.813465    2586 scope.go:117] "RemoveContainer" containerID="3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35"
Jul 29 00:48:18.813845 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:48:18.813805    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35\": container with ID starting with 3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35 not found: ID does not exist" containerID="3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35"
Jul 29 00:48:18.813930 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.813861    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35"} err="failed to get container status \"3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35\": rpc error: code = NotFound desc = could not find container \"3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35\": container with ID starting with 3d911ef2f490ed9ebf7f5a623fecde46db9f38fa7e7ce1e911b7f9058c840b35 not found: ID does not exist"
Jul 29 00:48:18.826259 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.826226    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d5bbd879f-59spd"]
Jul 29 00:48:18.831112 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:18.831082    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d5bbd879f-59spd"]
Jul 29 00:48:20.222976 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:48:20.222944    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d82b750-a0f6-4a3d-b8b8-00e791d1275a" path="/var/lib/kubelet/pods/0d82b750-a0f6-4a3d-b8b8-00e791d1275a/volumes"
Jul 29 00:50:23.333522 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:23.333481    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:50:24.913098 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:24.913051    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787f676965-kjbrt"]
Jul 29 00:50:24.913536 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:24.913517    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d82b750-a0f6-4a3d-b8b8-00e791d1275a" containerName="authorino"
Jul 29 00:50:24.913584 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:24.913539    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d82b750-a0f6-4a3d-b8b8-00e791d1275a" containerName="authorino"
Jul 29 00:50:24.913678 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:24.913664    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d82b750-a0f6-4a3d-b8b8-00e791d1275a" containerName="authorino"
Jul 29 00:50:24.916779 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:24.916759    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:50:24.920563 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:24.920538    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 00:50:24.920710 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:24.920574    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xnhp8\""
Jul 29 00:50:24.920710 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:24.920592    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 00:50:24.926966 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:24.926939    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787f676965-kjbrt"]
Jul 29 00:50:25.008810 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.008771    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgwq\" (UniqueName: \"kubernetes.io/projected/d222f951-af1d-4bff-8c93-5a7225cdba9a-kube-api-access-4fgwq\") pod \"maas-controller-787f676965-kjbrt\" (UID: \"d222f951-af1d-4bff-8c93-5a7225cdba9a\") " pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:50:25.008992 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.008824    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d222f951-af1d-4bff-8c93-5a7225cdba9a-webhook-cert\") pod \"maas-controller-787f676965-kjbrt\" (UID: \"d222f951-af1d-4bff-8c93-5a7225cdba9a\") " pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:50:25.110211 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.110157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgwq\" (UniqueName: \"kubernetes.io/projected/d222f951-af1d-4bff-8c93-5a7225cdba9a-kube-api-access-4fgwq\") pod \"maas-controller-787f676965-kjbrt\" (UID: \"d222f951-af1d-4bff-8c93-5a7225cdba9a\") " pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:50:25.110374 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.110320    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d222f951-af1d-4bff-8c93-5a7225cdba9a-webhook-cert\") pod \"maas-controller-787f676965-kjbrt\" (UID: \"d222f951-af1d-4bff-8c93-5a7225cdba9a\") " pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:50:25.112736 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.112712    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d222f951-af1d-4bff-8c93-5a7225cdba9a-webhook-cert\") pod \"maas-controller-787f676965-kjbrt\" (UID: \"d222f951-af1d-4bff-8c93-5a7225cdba9a\") " pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:50:25.119330 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.119295    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgwq\" (UniqueName: \"kubernetes.io/projected/d222f951-af1d-4bff-8c93-5a7225cdba9a-kube-api-access-4fgwq\") pod \"maas-controller-787f676965-kjbrt\" (UID: \"d222f951-af1d-4bff-8c93-5a7225cdba9a\") " pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:50:25.227964 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.227875    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:50:25.250537 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.250502    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:50:25.366783 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.365952    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787f676965-kjbrt"]
Jul 29 00:50:25.487337 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:25.487245    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787f676965-kjbrt" event={"ID":"d222f951-af1d-4bff-8c93-5a7225cdba9a","Type":"ContainerStarted","Data":"4eaaad23ab7e8b027624e52e14aabde5150daa03616a8ee0f759a0dfa23c7a91"}
Jul 29 00:50:26.494077 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:26.494039    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787f676965-kjbrt" event={"ID":"d222f951-af1d-4bff-8c93-5a7225cdba9a","Type":"ContainerStarted","Data":"198974d2c79a7035ba9d6bf828aa593c1a5f6d8bd39804d693633fe59995f11d"}
Jul 29 00:50:26.494529 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:26.494189    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:50:26.513883 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:26.513821    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787f676965-kjbrt" podStartSLOduration=2.083532404 podStartE2EDuration="2.513805668s" totalImagesPullingTime="430.273264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:50:26.512161036 +0000 UTC m=+856.821088881" watchObservedRunningTime="2026-07-29 00:50:26.513805668 +0000 UTC m=+856.822733509"
Jul 29 00:50:37.502515 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:50:37.502486    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787f676965-kjbrt"
Jul 29 00:51:13.107877 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:13.107836    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:14.395999 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:14.395966    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:18.999553 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:18.999511    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:19.103393 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:19.103354    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:19.200759 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:19.200722    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:19.353485 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:19.353453    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:25.029894 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:25.029860    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:25.109903 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:25.109866    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:30.773649 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:30.773591    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:30.825833 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:30.825801    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:31.048093 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:31.047996    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:51:31.104166 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:51:31.104133    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:52:30.988097 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:52:30.988060    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:52:31.043216 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:52:31.043168    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:54:54.153019 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:54:54.152986    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:55:34.146480 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:34.146379    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:55:37.640812 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:37.640774    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"]
Jul 29 00:55:37.642441 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:37.641443    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5" podUID="0c6af2a8-9cbd-45e7-be67-a1a756e18d09" containerName="manager" containerID="cri-o://19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92" gracePeriod=10
Jul 29 00:55:40.504580 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:40.504553    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:55:40.620288 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:40.620249    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9fzj\" (UniqueName: \"kubernetes.io/projected/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-kube-api-access-m9fzj\") pod \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\" (UID: \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\") "
Jul 29 00:55:40.620500 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:40.620345    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-extensions-socket-volume\") pod \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\" (UID: \"0c6af2a8-9cbd-45e7-be67-a1a756e18d09\") "
Jul 29 00:55:40.620773 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:40.620739    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0c6af2a8-9cbd-45e7-be67-a1a756e18d09" (UID: "0c6af2a8-9cbd-45e7-be67-a1a756e18d09"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:55:40.622465 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:40.622439    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-kube-api-access-m9fzj" (OuterVolumeSpecName: "kube-api-access-m9fzj") pod "0c6af2a8-9cbd-45e7-be67-a1a756e18d09" (UID: "0c6af2a8-9cbd-45e7-be67-a1a756e18d09"). InnerVolumeSpecName "kube-api-access-m9fzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:55:40.721680 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:40.721637    2586 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-extensions-socket-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:55:40.721680 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:40.721671    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fzj\" (UniqueName: \"kubernetes.io/projected/0c6af2a8-9cbd-45e7-be67-a1a756e18d09-kube-api-access-m9fzj\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 00:55:41.193590 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:41.193554    2586 generic.go:356] "Generic (PLEG): container finished" podID="0c6af2a8-9cbd-45e7-be67-a1a756e18d09" containerID="19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92" exitCode=0
Jul 29 00:55:41.193756 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:41.193595    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"
Jul 29 00:55:41.193756 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:41.193642    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5" event={"ID":"0c6af2a8-9cbd-45e7-be67-a1a756e18d09","Type":"ContainerDied","Data":"19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92"}
Jul 29 00:55:41.193756 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:41.193692    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5" event={"ID":"0c6af2a8-9cbd-45e7-be67-a1a756e18d09","Type":"ContainerDied","Data":"de5eeb5e89b6ec30e870d4bbd37213b2797d2ff4214adececfa0562fc343b684"}
Jul 29 00:55:41.193756 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:41.193722    2586 scope.go:117] "RemoveContainer" containerID="19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92"
Jul 29 00:55:41.205787 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:41.205771    2586 scope.go:117] "RemoveContainer" containerID="19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92"
Jul 29 00:55:41.206060 ip-10-0-132-227 kubenswrapper[2586]: E0729 00:55:41.206031    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92\": container with ID starting with 19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92 not found: ID does not exist" containerID="19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92"
Jul 29 00:55:41.206187 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:41.206066    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92"} err="failed to get container status \"19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92\": rpc error: code = NotFound desc = could not find container \"19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92\": container with ID starting with 19a5ce6deb155513eed63ac91a2f35c162ee9cb6b6ac90220b20cea34fe98b92 not found: ID does not exist"
Jul 29 00:55:41.224445 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:41.224423    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"]
Jul 29 00:55:41.226695 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:41.226676    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lpvm5"]
Jul 29 00:55:42.221583 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:55:42.221541    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6af2a8-9cbd-45e7-be67-a1a756e18d09" path="/var/lib/kubelet/pods/0c6af2a8-9cbd-45e7-be67-a1a756e18d09/volumes"
Jul 29 00:56:38.734844 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.734801    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"]
Jul 29 00:56:38.737400 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.735385    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6af2a8-9cbd-45e7-be67-a1a756e18d09" containerName="manager"
Jul 29 00:56:38.737400 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.735404    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6af2a8-9cbd-45e7-be67-a1a756e18d09" containerName="manager"
Jul 29 00:56:38.737400 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.735459    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6af2a8-9cbd-45e7-be67-a1a756e18d09" containerName="manager"
Jul 29 00:56:38.738391 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.738369    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:56:38.741127 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.741103    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sq5mr\""
Jul 29 00:56:38.751160 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.751135    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"]
Jul 29 00:56:38.812943 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.812896    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/386d3ba9-c326-4224-9c08-d3104525dc0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ggbbl\" (UID: \"386d3ba9-c326-4224-9c08-d3104525dc0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:56:38.813130 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.812981    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vch7j\" (UniqueName: \"kubernetes.io/projected/386d3ba9-c326-4224-9c08-d3104525dc0a-kube-api-access-vch7j\") pod \"kuadrant-operator-controller-manager-86878c769f-ggbbl\" (UID: \"386d3ba9-c326-4224-9c08-d3104525dc0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:56:38.913394 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.913353    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/386d3ba9-c326-4224-9c08-d3104525dc0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ggbbl\" (UID: \"386d3ba9-c326-4224-9c08-d3104525dc0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:56:38.913554 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.913465    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vch7j\" (UniqueName: \"kubernetes.io/projected/386d3ba9-c326-4224-9c08-d3104525dc0a-kube-api-access-vch7j\") pod \"kuadrant-operator-controller-manager-86878c769f-ggbbl\" (UID: \"386d3ba9-c326-4224-9c08-d3104525dc0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:56:38.913727 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.913707    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/386d3ba9-c326-4224-9c08-d3104525dc0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ggbbl\" (UID: \"386d3ba9-c326-4224-9c08-d3104525dc0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:56:38.921987 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:38.921964    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vch7j\" (UniqueName: \"kubernetes.io/projected/386d3ba9-c326-4224-9c08-d3104525dc0a-kube-api-access-vch7j\") pod \"kuadrant-operator-controller-manager-86878c769f-ggbbl\" (UID: \"386d3ba9-c326-4224-9c08-d3104525dc0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:56:39.048771 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:39.048678    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:56:39.193481 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:39.193454    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"]
Jul 29 00:56:39.195407 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:56:39.195376    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386d3ba9_c326_4224_9c08_d3104525dc0a.slice/crio-e081e4c341ab2309037fb987fcc26b6b8687e5a5cdb3b45fdd5f9146863e616c WatchSource:0}: Error finding container e081e4c341ab2309037fb987fcc26b6b8687e5a5cdb3b45fdd5f9146863e616c: Status 404 returned error can't find the container with id e081e4c341ab2309037fb987fcc26b6b8687e5a5cdb3b45fdd5f9146863e616c
Jul 29 00:56:39.505459 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:39.505425    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl" event={"ID":"386d3ba9-c326-4224-9c08-d3104525dc0a","Type":"ContainerStarted","Data":"fb9cd26a02ee2162cbdb4147a903e03672592e6644e055297d85e8b03d6bc2c3"}
Jul 29 00:56:39.505676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:39.505479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl" event={"ID":"386d3ba9-c326-4224-9c08-d3104525dc0a","Type":"ContainerStarted","Data":"e081e4c341ab2309037fb987fcc26b6b8687e5a5cdb3b45fdd5f9146863e616c"}
Jul 29 00:56:39.505676 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:39.505550    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:56:39.522293 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:39.522236    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl" podStartSLOduration=1.522219853 podStartE2EDuration="1.522219853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:56:39.521776138 +0000 UTC m=+1229.830703979" watchObservedRunningTime="2026-07-29 00:56:39.522219853 +0000 UTC m=+1229.831147692"
Jul 29 00:56:50.513167 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:56:50.513135    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ggbbl"
Jul 29 00:57:47.375322 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:57:47.375285    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:59:29.316760 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.316725    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"]
Jul 29 00:59:29.320305 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.320281    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.323251 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.323188    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-dc2cca86-kserve-self-signed-certs\""
Jul 29 00:59:29.341697 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.341668    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"]
Jul 29 00:59:29.377055 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.377025    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:59:29.415967 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.415928    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-model-cache\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.416143 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.415978    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-home\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.416143 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.416075    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zfkz\" (UniqueName: \"kubernetes.io/projected/bac93686-c424-4a12-8a3d-d6b70abb81a3-kube-api-access-5zfkz\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.416250 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.416155    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-dshm\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.416250 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.416189    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-tmp-dir\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.416250 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.416231    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac93686-c424-4a12-8a3d-d6b70abb81a3-tls-certs\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.517011 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.516968    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-model-cache\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.517254 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.517021    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-home\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.517254 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.517060    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zfkz\" (UniqueName: \"kubernetes.io/projected/bac93686-c424-4a12-8a3d-d6b70abb81a3-kube-api-access-5zfkz\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.517254 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.517097    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-dshm\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.517254 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.517175    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-tmp-dir\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.517254 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.517242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac93686-c424-4a12-8a3d-d6b70abb81a3-tls-certs\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.517539 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.517477    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-model-cache\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.517584 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.517544    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-home\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.517584 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.517564    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-tmp-dir\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.519589 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.519561    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-dshm\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.519839 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.519820    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac93686-c424-4a12-8a3d-d6b70abb81a3-tls-certs\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.535036 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.535006    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zfkz\" (UniqueName: \"kubernetes.io/projected/bac93686-c424-4a12-8a3d-d6b70abb81a3-kube-api-access-5zfkz\") pod \"e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") " pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.631131 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.631049    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:29.772826 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:29.772792    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"]
Jul 29 00:59:29.774456 ip-10-0-132-227 kubenswrapper[2586]: W0729 00:59:29.774424    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbac93686_c424_4a12_8a3d_d6b70abb81a3.slice/crio-b283f2a9c8d59cb3020ba4b67a0ec5f0719e20fc8435e25966ab11b7f1c331ca WatchSource:0}: Error finding container b283f2a9c8d59cb3020ba4b67a0ec5f0719e20fc8435e25966ab11b7f1c331ca: Status 404 returned error can't find the container with id b283f2a9c8d59cb3020ba4b67a0ec5f0719e20fc8435e25966ab11b7f1c331ca
Jul 29 00:59:30.439516 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:30.439479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9" event={"ID":"bac93686-c424-4a12-8a3d-d6b70abb81a3","Type":"ContainerStarted","Data":"40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd"}
Jul 29 00:59:30.439516 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:30.439522    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9" event={"ID":"bac93686-c424-4a12-8a3d-d6b70abb81a3","Type":"ContainerStarted","Data":"b283f2a9c8d59cb3020ba4b67a0ec5f0719e20fc8435e25966ab11b7f1c331ca"}
Jul 29 00:59:30.459427 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:30.459363    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9" podStartSLOduration=1.459339847 podStartE2EDuration="1.459339847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:59:30.45591101 +0000 UTC m=+1400.764838853" watchObservedRunningTime="2026-07-29 00:59:30.459339847 +0000 UTC m=+1400.768267690"
Jul 29 00:59:38.835170 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:38.835136    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 00:59:39.631749 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:39.631715    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:39.631964 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:39.631796    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:39.644325 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:39.644303    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 00:59:40.503058 ip-10-0-132-227 kubenswrapper[2586]: I0729 00:59:40.503029    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 01:00:00.142232 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:00.142159    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn"]
Jul 29 01:00:00.146420 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:00.146391    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn"
Jul 29 01:00:00.149715 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:00.149681    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:00:00.152217 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:00.152175    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn"]
Jul 29 01:00:00.159189 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:00.159163    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn"
Jul 29 01:00:00.301761 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:00.301731    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn"]
Jul 29 01:00:00.303540 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:00:00.303507    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131ddd08_2d3b_4a59_be83_acc69e45df2a.slice/crio-0b9f12d75922a212de42df444d9cb6a6f7a8cee319f19f6b6b85c95e2e6451c9 WatchSource:0}: Error finding container 0b9f12d75922a212de42df444d9cb6a6f7a8cee319f19f6b6b85c95e2e6451c9: Status 404 returned error can't find the container with id 0b9f12d75922a212de42df444d9cb6a6f7a8cee319f19f6b6b85c95e2e6451c9
Jul 29 01:00:00.305567 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:00.305549    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:00:00.605110 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:00.605067    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn" event={"ID":"131ddd08-2d3b-4a59-be83-acc69e45df2a","Type":"ContainerStarted","Data":"0b9f12d75922a212de42df444d9cb6a6f7a8cee319f19f6b6b85c95e2e6451c9"}
Jul 29 01:00:03.624475 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:03.624433    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn" event={"ID":"131ddd08-2d3b-4a59-be83-acc69e45df2a","Type":"ContainerStarted","Data":"e17da10bb219578db2fe0378784dcb1b0931a35bc73eec7c18b875205f0d331b"}
Jul 29 01:00:03.641941 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:03.641889    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn" podStartSLOduration=1.018631228 podStartE2EDuration="3.641873328s" totalImagesPullingTime="2.6232421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:00:03.639495448 +0000 UTC m=+1433.948423290" watchObservedRunningTime="2026-07-29 01:00:03.641873328 +0000 UTC m=+1433.950801213"
Jul 29 01:00:04.630855 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:04.630823    2586 generic.go:356] "Generic (PLEG): container finished" podID="131ddd08-2d3b-4a59-be83-acc69e45df2a" containerID="e17da10bb219578db2fe0378784dcb1b0931a35bc73eec7c18b875205f0d331b" exitCode=0
Jul 29 01:00:04.630855 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:04.630859    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn" event={"ID":"131ddd08-2d3b-4a59-be83-acc69e45df2a","Type":"ContainerDied","Data":"e17da10bb219578db2fe0378784dcb1b0931a35bc73eec7c18b875205f0d331b"}
Jul 29 01:00:05.769165 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:05.769139    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn"
Jul 29 01:00:06.643633 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:06.643604    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn"
Jul 29 01:00:06.643804 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:06.643595    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754780-pjppn" event={"ID":"131ddd08-2d3b-4a59-be83-acc69e45df2a","Type":"ContainerDied","Data":"0b9f12d75922a212de42df444d9cb6a6f7a8cee319f19f6b6b85c95e2e6451c9"}
Jul 29 01:00:06.643804 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:06.643726    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9f12d75922a212de42df444d9cb6a6f7a8cee319f19f6b6b85c95e2e6451c9"
Jul 29 01:00:16.136990 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:16.136953    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"]
Jul 29 01:00:17.095653 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:17.095620    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:00:17.197131 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:17.197093    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:00:31.158000 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.157956    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9" podUID="bac93686-c424-4a12-8a3d-d6b70abb81a3" containerName="main" containerID="cri-o://40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd" gracePeriod=45
Jul 29 01:00:31.411529 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.411457    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 01:00:31.472403 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.472359    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zfkz\" (UniqueName: \"kubernetes.io/projected/bac93686-c424-4a12-8a3d-d6b70abb81a3-kube-api-access-5zfkz\") pod \"bac93686-c424-4a12-8a3d-d6b70abb81a3\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") "
Jul 29 01:00:31.472577 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.472421    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac93686-c424-4a12-8a3d-d6b70abb81a3-tls-certs\") pod \"bac93686-c424-4a12-8a3d-d6b70abb81a3\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") "
Jul 29 01:00:31.472577 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.472450    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-home\") pod \"bac93686-c424-4a12-8a3d-d6b70abb81a3\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") "
Jul 29 01:00:31.472664 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.472632    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-dshm\") pod \"bac93686-c424-4a12-8a3d-d6b70abb81a3\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") "
Jul 29 01:00:31.472717 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.472668    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-tmp-dir\") pod \"bac93686-c424-4a12-8a3d-d6b70abb81a3\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") "
Jul 29 01:00:31.472717 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.472696    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-model-cache\") pod \"bac93686-c424-4a12-8a3d-d6b70abb81a3\" (UID: \"bac93686-c424-4a12-8a3d-d6b70abb81a3\") "
Jul 29 01:00:31.472829 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.472738    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-home" (OuterVolumeSpecName: "home") pod "bac93686-c424-4a12-8a3d-d6b70abb81a3" (UID: "bac93686-c424-4a12-8a3d-d6b70abb81a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:00:31.472955 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.472924    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bac93686-c424-4a12-8a3d-d6b70abb81a3" (UID: "bac93686-c424-4a12-8a3d-d6b70abb81a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:00:31.473068 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.473038    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-tmp-dir\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:00:31.473068 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.473036    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "bac93686-c424-4a12-8a3d-d6b70abb81a3" (UID: "bac93686-c424-4a12-8a3d-d6b70abb81a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:00:31.473068 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.473055    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-home\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:00:31.474810 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.474773    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-dshm" (OuterVolumeSpecName: "dshm") pod "bac93686-c424-4a12-8a3d-d6b70abb81a3" (UID: "bac93686-c424-4a12-8a3d-d6b70abb81a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:00:31.474810 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.474780    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac93686-c424-4a12-8a3d-d6b70abb81a3-kube-api-access-5zfkz" (OuterVolumeSpecName: "kube-api-access-5zfkz") pod "bac93686-c424-4a12-8a3d-d6b70abb81a3" (UID: "bac93686-c424-4a12-8a3d-d6b70abb81a3"). InnerVolumeSpecName "kube-api-access-5zfkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:00:31.474968 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.474883    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac93686-c424-4a12-8a3d-d6b70abb81a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bac93686-c424-4a12-8a3d-d6b70abb81a3" (UID: "bac93686-c424-4a12-8a3d-d6b70abb81a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:00:31.573852 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.573817    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-dshm\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:00:31.573852 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.573850    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac93686-c424-4a12-8a3d-d6b70abb81a3-model-cache\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:00:31.574070 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.573867    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zfkz\" (UniqueName: \"kubernetes.io/projected/bac93686-c424-4a12-8a3d-d6b70abb81a3-kube-api-access-5zfkz\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:00:31.574070 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.573881    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac93686-c424-4a12-8a3d-d6b70abb81a3-tls-certs\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:00:31.787797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.787699    2586 generic.go:356] "Generic (PLEG): container finished" podID="bac93686-c424-4a12-8a3d-d6b70abb81a3" containerID="40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd" exitCode=0
Jul 29 01:00:31.787797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.787744    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9" event={"ID":"bac93686-c424-4a12-8a3d-d6b70abb81a3","Type":"ContainerDied","Data":"40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd"}
Jul 29 01:00:31.787797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.787749    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"
Jul 29 01:00:31.787797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.787776    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9" event={"ID":"bac93686-c424-4a12-8a3d-d6b70abb81a3","Type":"ContainerDied","Data":"b283f2a9c8d59cb3020ba4b67a0ec5f0719e20fc8435e25966ab11b7f1c331ca"}
Jul 29 01:00:31.787797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.787798    2586 scope.go:117] "RemoveContainer" containerID="40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd"
Jul 29 01:00:31.797511 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.797487    2586 scope.go:117] "RemoveContainer" containerID="40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd"
Jul 29 01:00:31.797761 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:00:31.797737    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd\": container with ID starting with 40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd not found: ID does not exist" containerID="40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd"
Jul 29 01:00:31.797813 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.797771    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd"} err="failed to get container status \"40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd\": rpc error: code = NotFound desc = could not find container \"40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd\": container with ID starting with 40c1cbbb627a4159aa1fb4b6f4a82a5d3d842a35d83cef7f816b1a245be21fcd not found: ID does not exist"
Jul 29 01:00:31.810766 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.810736    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"]
Jul 29 01:00:31.814051 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:31.814024    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-dc2cca86-kserve-89fb66b75-r4hc9"]
Jul 29 01:00:32.221680 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:00:32.221646    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac93686-c424-4a12-8a3d-d6b70abb81a3" path="/var/lib/kubelet/pods/bac93686-c424-4a12-8a3d-d6b70abb81a3/volumes"
Jul 29 01:02:34.815346 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:02:34.815301    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:02:46.147222 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:02:46.147165    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:25.703026 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:25.702990    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:47.182909 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:47.182871    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:50.168693 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:50.168659    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:50.268532 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:50.268497    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:50.571141 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:50.571111    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:51.016540 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:51.016451    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:56.416576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:56.416539    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:56.486100 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:56.486070    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:56.565622 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:56.565588    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:56.873474 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:56.873437    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:56.997646 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:56.997617    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:58.940885 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:58.940853    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:59.040967 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.040935    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:59.187554 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.187515    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:59.377465 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.377430    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:59.866410 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.866373    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:03:59.971477 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.971444    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"]
Jul 29 01:03:59.971860 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.971812    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac93686-c424-4a12-8a3d-d6b70abb81a3" containerName="main"
Jul 29 01:03:59.971860 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.971823    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac93686-c424-4a12-8a3d-d6b70abb81a3" containerName="main"
Jul 29 01:03:59.971860 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.971835    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131ddd08-2d3b-4a59-be83-acc69e45df2a" containerName="cleanup"
Jul 29 01:03:59.971860 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.971841    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="131ddd08-2d3b-4a59-be83-acc69e45df2a" containerName="cleanup"
Jul 29 01:03:59.971997 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.971908    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac93686-c424-4a12-8a3d-d6b70abb81a3" containerName="main"
Jul 29 01:03:59.971997 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.971917    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="131ddd08-2d3b-4a59-be83-acc69e45df2a" containerName="cleanup"
Jul 29 01:03:59.975053 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.975032    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:03:59.978205 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.978168    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f6bbaf0d-dockercfg-7d85s\""
Jul 29 01:03:59.978339 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.978256    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f6bbaf0d\""
Jul 29 01:03:59.986710 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.986684    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"]
Jul 29 01:03:59.998756 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:03:59.998723    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"]
Jul 29 01:04:00.003061 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.003036    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:00.013611 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.013573    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"]
Jul 29 01:04:00.080265 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.080228    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:00.131849 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.131742    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8217f88-7b30-411e-9841-22c135773b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5\" (UID: \"b8217f88-7b30-411e-9841-22c135773b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:00.131849 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.131805    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62z6b\" (UniqueName: \"kubernetes.io/projected/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-kube-api-access-62z6b\") pod \"payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl\" (UID: \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\") " pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:00.131849 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.131830    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4b4\" (UniqueName: \"kubernetes.io/projected/b8217f88-7b30-411e-9841-22c135773b2f-kube-api-access-rp4b4\") pod \"payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5\" (UID: \"b8217f88-7b30-411e-9841-22c135773b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:00.132078 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.131930    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl\" (UID: \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\") " pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:00.232434 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.232399    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4b4\" (UniqueName: \"kubernetes.io/projected/b8217f88-7b30-411e-9841-22c135773b2f-kube-api-access-rp4b4\") pod \"payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5\" (UID: \"b8217f88-7b30-411e-9841-22c135773b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:00.232610 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.232480    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl\" (UID: \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\") " pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:00.232610 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.232550    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8217f88-7b30-411e-9841-22c135773b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5\" (UID: \"b8217f88-7b30-411e-9841-22c135773b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:00.232610 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.232601    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62z6b\" (UniqueName: \"kubernetes.io/projected/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-kube-api-access-62z6b\") pod \"payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl\" (UID: \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\") " pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:00.233456 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.233431    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8217f88-7b30-411e-9841-22c135773b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5\" (UID: \"b8217f88-7b30-411e-9841-22c135773b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:00.233511 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.233431    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl\" (UID: \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\") " pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:00.241053 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.241024    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62z6b\" (UniqueName: \"kubernetes.io/projected/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-kube-api-access-62z6b\") pod \"payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl\" (UID: \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\") " pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:00.241263 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.241147    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4b4\" (UniqueName: \"kubernetes.io/projected/b8217f88-7b30-411e-9841-22c135773b2f-kube-api-access-rp4b4\") pod \"payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5\" (UID: \"b8217f88-7b30-411e-9841-22c135773b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:00.287394 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.287361    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:00.319635 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.319604    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:00.447626 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.447595    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"]
Jul 29 01:04:00.450267 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:04:00.450231    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8217f88_7b30_411e_9841_22c135773b2f.slice/crio-d16b4f8ccfb430d96be72ce0237be00cb66ec02abee3af4379053c4f61cac1f0 WatchSource:0}: Error finding container d16b4f8ccfb430d96be72ce0237be00cb66ec02abee3af4379053c4f61cac1f0: Status 404 returned error can't find the container with id d16b4f8ccfb430d96be72ce0237be00cb66ec02abee3af4379053c4f61cac1f0
Jul 29 01:04:00.491110 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.491080    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"]
Jul 29 01:04:00.492072 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:04:00.492044    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8bf502_4bd2_42de_b4e8_b5f0ce5cb98f.slice/crio-8fb231c3a2ae6186bd93f70ae3d4a2bec39cf801d15b9d9b6c05bf3d85e64ab4 WatchSource:0}: Error finding container 8fb231c3a2ae6186bd93f70ae3d4a2bec39cf801d15b9d9b6c05bf3d85e64ab4: Status 404 returned error can't find the container with id 8fb231c3a2ae6186bd93f70ae3d4a2bec39cf801d15b9d9b6c05bf3d85e64ab4
Jul 29 01:04:00.951378 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.951332    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5" event={"ID":"b8217f88-7b30-411e-9841-22c135773b2f","Type":"ContainerStarted","Data":"390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4"}
Jul 29 01:04:00.951612 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.951449    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5" event={"ID":"b8217f88-7b30-411e-9841-22c135773b2f","Type":"ContainerStarted","Data":"d16b4f8ccfb430d96be72ce0237be00cb66ec02abee3af4379053c4f61cac1f0"}
Jul 29 01:04:00.951612 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.951483    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:00.953110 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.953078    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl" event={"ID":"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f","Type":"ContainerStarted","Data":"2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7"}
Jul 29 01:04:00.953283 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.953135    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl" event={"ID":"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f","Type":"ContainerStarted","Data":"8fb231c3a2ae6186bd93f70ae3d4a2bec39cf801d15b9d9b6c05bf3d85e64ab4"}
Jul 29 01:04:00.953283 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.953165    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:00.968765 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.968700    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5" podStartSLOduration=1.9686776350000001 podStartE2EDuration="1.968677635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:04:00.965587254 +0000 UTC m=+1671.274515239" watchObservedRunningTime="2026-07-29 01:04:00.968677635 +0000 UTC m=+1671.277605483"
Jul 29 01:04:00.983387 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:00.983330    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl" podStartSLOduration=1.9833142430000001 podStartE2EDuration="1.983314243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:04:00.981834577 +0000 UTC m=+1671.290762423" watchObservedRunningTime="2026-07-29 01:04:00.983314243 +0000 UTC m=+1671.292242084"
Jul 29 01:04:11.961016 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:11.960981    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:11.961423 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:11.961146    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:24.486285 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.486241    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"]
Jul 29 01:04:24.489872 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.489847    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"
Jul 29 01:04:24.492471 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.492446    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:04:24.492578 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.492475    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:04:24.496934 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.496912    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"]
Jul 29 01:04:24.547973 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.547931    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10329180-4b41-4222-b811-8718d56199d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4\" (UID: \"10329180-4b41-4222-b811-8718d56199d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"
Jul 29 01:04:24.649077 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.649033    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10329180-4b41-4222-b811-8718d56199d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4\" (UID: \"10329180-4b41-4222-b811-8718d56199d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"
Jul 29 01:04:24.649706 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.649682    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10329180-4b41-4222-b811-8718d56199d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4\" (UID: \"10329180-4b41-4222-b811-8718d56199d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"
Jul 29 01:04:24.802443 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.802350    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"
Jul 29 01:04:24.941857 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:24.941827    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"]
Jul 29 01:04:24.943895 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:04:24.943866    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10329180_4b41_4222_b811_8718d56199d0.slice/crio-50b2ca3de8fc2514f53e1aad71b1bedea14a1e8e12426c8f5a59414b47f30a1c WatchSource:0}: Error finding container 50b2ca3de8fc2514f53e1aad71b1bedea14a1e8e12426c8f5a59414b47f30a1c: Status 404 returned error can't find the container with id 50b2ca3de8fc2514f53e1aad71b1bedea14a1e8e12426c8f5a59414b47f30a1c
Jul 29 01:04:25.093117 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:25.093079    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4" event={"ID":"10329180-4b41-4222-b811-8718d56199d0","Type":"ContainerStarted","Data":"50b2ca3de8fc2514f53e1aad71b1bedea14a1e8e12426c8f5a59414b47f30a1c"}
Jul 29 01:04:26.100814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:26.100776    2586 generic.go:356] "Generic (PLEG): container finished" podID="10329180-4b41-4222-b811-8718d56199d0" containerID="1480fa8d80fe3c5a16cf51252eea26706a847c27ae065057decf0246686c93d4" exitCode=0
Jul 29 01:04:26.101258 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:26.100865    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4" event={"ID":"10329180-4b41-4222-b811-8718d56199d0","Type":"ContainerDied","Data":"1480fa8d80fe3c5a16cf51252eea26706a847c27ae065057decf0246686c93d4"}
Jul 29 01:04:27.248599 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:27.248571    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"
Jul 29 01:04:27.271415 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:27.271373    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10329180-4b41-4222-b811-8718d56199d0-maas-api-service-ca\") pod \"10329180-4b41-4222-b811-8718d56199d0\" (UID: \"10329180-4b41-4222-b811-8718d56199d0\") "
Jul 29 01:04:27.271800 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:27.271776    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10329180-4b41-4222-b811-8718d56199d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10329180-4b41-4222-b811-8718d56199d0" (UID: "10329180-4b41-4222-b811-8718d56199d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:04:27.372726 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:27.372644    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10329180-4b41-4222-b811-8718d56199d0-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:04:28.112873 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:28.112839    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4" event={"ID":"10329180-4b41-4222-b811-8718d56199d0","Type":"ContainerDied","Data":"50b2ca3de8fc2514f53e1aad71b1bedea14a1e8e12426c8f5a59414b47f30a1c"}
Jul 29 01:04:28.112873 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:28.112868    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"
Jul 29 01:04:28.113164 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:28.112890    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50b2ca3de8fc2514f53e1aad71b1bedea14a1e8e12426c8f5a59414b47f30a1c"
Jul 29 01:04:29.550835 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.550807    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:29.573269 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.573224    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"]
Jul 29 01:04:29.573502 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.573470    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl" podUID="4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f" containerName="payload-processing" containerID="cri-o://2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7" gracePeriod=30
Jul 29 01:04:29.579077 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.579053    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"]
Jul 29 01:04:29.579299 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.579274    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5" podUID="b8217f88-7b30-411e-9841-22c135773b2f" containerName="payload-pre-processing" containerID="cri-o://390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4" gracePeriod=30
Jul 29 01:04:29.617501 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.617468    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:29.709742 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.709710    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:29.716669 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.716635    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"]
Jul 29 01:04:29.720293 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.720264    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6bbaf0d-6qvs4"]
Jul 29 01:04:29.767762 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.767694    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:29.842934 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.842887    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:29.867011 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.866969    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:29.882559 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.882525    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:29.895577 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.895550    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62z6b\" (UniqueName: \"kubernetes.io/projected/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-kube-api-access-62z6b\") pod \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\" (UID: \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\") "
Jul 29 01:04:29.895713 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.895608    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp4b4\" (UniqueName: \"kubernetes.io/projected/b8217f88-7b30-411e-9841-22c135773b2f-kube-api-access-rp4b4\") pod \"b8217f88-7b30-411e-9841-22c135773b2f\" (UID: \"b8217f88-7b30-411e-9841-22c135773b2f\") "
Jul 29 01:04:29.895713 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.895637    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-plugins-config-volume\") pod \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\" (UID: \"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f\") "
Jul 29 01:04:29.895713 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.895663    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8217f88-7b30-411e-9841-22c135773b2f-plugins-config-volume\") pod \"b8217f88-7b30-411e-9841-22c135773b2f\" (UID: \"b8217f88-7b30-411e-9841-22c135773b2f\") "
Jul 29 01:04:29.896048 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.896005    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f" (UID: "4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:04:29.896169 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.896062    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8217f88-7b30-411e-9841-22c135773b2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8217f88-7b30-411e-9841-22c135773b2f" (UID: "b8217f88-7b30-411e-9841-22c135773b2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:04:29.897623 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.897603    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8217f88-7b30-411e-9841-22c135773b2f-kube-api-access-rp4b4" (OuterVolumeSpecName: "kube-api-access-rp4b4") pod "b8217f88-7b30-411e-9841-22c135773b2f" (UID: "b8217f88-7b30-411e-9841-22c135773b2f"). InnerVolumeSpecName "kube-api-access-rp4b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:04:29.897680 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.897652    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-kube-api-access-62z6b" (OuterVolumeSpecName: "kube-api-access-62z6b") pod "4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f" (UID: "4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f"). InnerVolumeSpecName "kube-api-access-62z6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:04:29.997089 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.997046    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62z6b\" (UniqueName: \"kubernetes.io/projected/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-kube-api-access-62z6b\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:04:29.997089 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.997082    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp4b4\" (UniqueName: \"kubernetes.io/projected/b8217f88-7b30-411e-9841-22c135773b2f-kube-api-access-rp4b4\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:04:29.997089 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.997092    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:04:29.997089 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:29.997102    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8217f88-7b30-411e-9841-22c135773b2f-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:04:30.079652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.079566    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:30.126364 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.126328    2586 generic.go:356] "Generic (PLEG): container finished" podID="b8217f88-7b30-411e-9841-22c135773b2f" containerID="390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4" exitCode=0
Jul 29 01:04:30.126546 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.126370    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"
Jul 29 01:04:30.126546 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.126411    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5" event={"ID":"b8217f88-7b30-411e-9841-22c135773b2f","Type":"ContainerDied","Data":"390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4"}
Jul 29 01:04:30.126546 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.126490    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5" event={"ID":"b8217f88-7b30-411e-9841-22c135773b2f","Type":"ContainerDied","Data":"d16b4f8ccfb430d96be72ce0237be00cb66ec02abee3af4379053c4f61cac1f0"}
Jul 29 01:04:30.126546 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.126514    2586 scope.go:117] "RemoveContainer" containerID="390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4"
Jul 29 01:04:30.127717 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.127699    2586 generic.go:356] "Generic (PLEG): container finished" podID="4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f" containerID="2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7" exitCode=0
Jul 29 01:04:30.127817 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.127767    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"
Jul 29 01:04:30.127817 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.127784    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl" event={"ID":"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f","Type":"ContainerDied","Data":"2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7"}
Jul 29 01:04:30.127923 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.127834    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl" event={"ID":"4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f","Type":"ContainerDied","Data":"8fb231c3a2ae6186bd93f70ae3d4a2bec39cf801d15b9d9b6c05bf3d85e64ab4"}
Jul 29 01:04:30.136580 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.136562    2586 scope.go:117] "RemoveContainer" containerID="390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4"
Jul 29 01:04:30.136869 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:04:30.136848    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4\": container with ID starting with 390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4 not found: ID does not exist" containerID="390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4"
Jul 29 01:04:30.136948 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.136883    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4"} err="failed to get container status \"390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4\": rpc error: code = NotFound desc = could not find container \"390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4\": container with ID starting with 390f8e8f6939b8bc52f79358972e2dbe77d1966747e95fb065b634650a513fe4 not found: ID does not exist"
Jul 29 01:04:30.136948 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.136906    2586 scope.go:117] "RemoveContainer" containerID="2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7"
Jul 29 01:04:30.145387 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.145367    2586 scope.go:117] "RemoveContainer" containerID="2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7"
Jul 29 01:04:30.145652 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:04:30.145633    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7\": container with ID starting with 2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7 not found: ID does not exist" containerID="2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7"
Jul 29 01:04:30.145698 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.145660    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7"} err="failed to get container status \"2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7\": rpc error: code = NotFound desc = could not find container \"2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7\": container with ID starting with 2bc21338fab2d4a9518a6aa11bf58f3316388bfeadbd160f76cae27149f781b7 not found: ID does not exist"
Jul 29 01:04:30.152684 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.152661    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"]
Jul 29 01:04:30.156247 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.156225    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6bbaf0d-75f88d54b8-l5mw5"]
Jul 29 01:04:30.166930 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.166910    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"]
Jul 29 01:04:30.170467 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.170437    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6bbaf0d-cff7b46b6-jpqvl"]
Jul 29 01:04:30.222403 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.222367    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10329180-4b41-4222-b811-8718d56199d0" path="/var/lib/kubelet/pods/10329180-4b41-4222-b811-8718d56199d0/volumes"
Jul 29 01:04:30.222918 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.222896    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f" path="/var/lib/kubelet/pods/4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f/volumes"
Jul 29 01:04:30.223291 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.223277    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8217f88-7b30-411e-9841-22c135773b2f" path="/var/lib/kubelet/pods/b8217f88-7b30-411e-9841-22c135773b2f/volumes"
Jul 29 01:04:30.311795 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.311764    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:30.412230 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.412124    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:30.550556 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.550521    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:30.860997 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:30.860963    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:31.051386 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:31.051357    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:33.450000 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.449903    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"]
Jul 29 01:04:33.450489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.450390    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8217f88-7b30-411e-9841-22c135773b2f" containerName="payload-pre-processing"
Jul 29 01:04:33.450489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.450406    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8217f88-7b30-411e-9841-22c135773b2f" containerName="payload-pre-processing"
Jul 29 01:04:33.450489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.450418    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10329180-4b41-4222-b811-8718d56199d0" containerName="revoke-keys"
Jul 29 01:04:33.450489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.450425    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="10329180-4b41-4222-b811-8718d56199d0" containerName="revoke-keys"
Jul 29 01:04:33.450489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.450435    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f" containerName="payload-processing"
Jul 29 01:04:33.450489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.450441    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f" containerName="payload-processing"
Jul 29 01:04:33.450815 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.450522    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8217f88-7b30-411e-9841-22c135773b2f" containerName="payload-pre-processing"
Jul 29 01:04:33.450815 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.450531    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="10329180-4b41-4222-b811-8718d56199d0" containerName="revoke-keys"
Jul 29 01:04:33.450815 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.450539    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8bf502-4bd2-42de-b4e8-b5f0ce5cb98f" containerName="payload-processing"
Jul 29 01:04:33.455084 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.455062    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.457710 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.457684    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-956869dd-serving-cert\""
Jul 29 01:04:33.457853 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.457814    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9jxwk\""
Jul 29 01:04:33.460624 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.459288    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:04:33.460624 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.459686    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 01:04:33.462887 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.462862    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"]
Jul 29 01:04:33.529287 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.529250    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f00ee30-2563-427d-b47a-61151e0d7fab-maas-api-tls\") pod \"maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.529482 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.529317    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5flt\" (UniqueName: \"kubernetes.io/projected/8f00ee30-2563-427d-b47a-61151e0d7fab-kube-api-access-w5flt\") pod \"maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.529482 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.529406    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f00ee30-2563-427d-b47a-61151e0d7fab-openshift-service-ca\") pod \"maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.629985 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.629946    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f00ee30-2563-427d-b47a-61151e0d7fab-openshift-service-ca\") pod \"maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.630175 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.630018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f00ee30-2563-427d-b47a-61151e0d7fab-maas-api-tls\") pod \"maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.630175 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.630069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5flt\" (UniqueName: \"kubernetes.io/projected/8f00ee30-2563-427d-b47a-61151e0d7fab-kube-api-access-w5flt\") pod \"maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.630642 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.630610    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f00ee30-2563-427d-b47a-61151e0d7fab-openshift-service-ca\") pod \"maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.632527 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.632507    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f00ee30-2563-427d-b47a-61151e0d7fab-maas-api-tls\") pod \"maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.638818 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.638785    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5flt\" (UniqueName: \"kubernetes.io/projected/8f00ee30-2563-427d-b47a-61151e0d7fab-kube-api-access-w5flt\") pod \"maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.650694 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.650668    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:33.752005 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.751917    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:33.771126 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.771094    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:33.923287 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:33.923255    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"]
Jul 29 01:04:33.925211 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:04:33.925168    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f00ee30_2563_427d_b47a_61151e0d7fab.slice/crio-cb8c7c3e0cdb4665ef16b96d9b5e8d3f40a66d0eb3e1666fc1d0a9c500d1c1cf WatchSource:0}: Error finding container cb8c7c3e0cdb4665ef16b96d9b5e8d3f40a66d0eb3e1666fc1d0a9c500d1c1cf: Status 404 returned error can't find the container with id cb8c7c3e0cdb4665ef16b96d9b5e8d3f40a66d0eb3e1666fc1d0a9c500d1c1cf
Jul 29 01:04:34.023708 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.023611    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:34.153007 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.152958    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz" event={"ID":"8f00ee30-2563-427d-b47a-61151e0d7fab","Type":"ContainerStarted","Data":"cb8c7c3e0cdb4665ef16b96d9b5e8d3f40a66d0eb3e1666fc1d0a9c500d1c1cf"}
Jul 29 01:04:34.169208 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.169165    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:04:34.199854 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.199820    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"]
Jul 29 01:04:34.204571 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.204549    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:34.207108 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.207090    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-956869dd\""
Jul 29 01:04:34.212914 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.212605    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"]
Jul 29 01:04:34.235601 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.235556    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl44v\" (UniqueName: \"kubernetes.io/projected/c4948d0f-71ec-46da-b5f9-df15a97ae835-kube-api-access-hl44v\") pod \"payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g\" (UID: \"c4948d0f-71ec-46da-b5f9-df15a97ae835\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:34.235774 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.235700    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4948d0f-71ec-46da-b5f9-df15a97ae835-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g\" (UID: \"c4948d0f-71ec-46da-b5f9-df15a97ae835\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:34.307621 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.307534    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"]
Jul 29 01:04:34.311158 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.311137    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:34.313915 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.313887    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-956869dd-dockercfg-2g5dg\""
Jul 29 01:04:34.322388 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.322356    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"]
Jul 29 01:04:34.336321 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.336267    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4948d0f-71ec-46da-b5f9-df15a97ae835-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g\" (UID: \"c4948d0f-71ec-46da-b5f9-df15a97ae835\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:34.336541 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.336422    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl44v\" (UniqueName: \"kubernetes.io/projected/c4948d0f-71ec-46da-b5f9-df15a97ae835-kube-api-access-hl44v\") pod \"payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g\" (UID: \"c4948d0f-71ec-46da-b5f9-df15a97ae835\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:34.336541 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.336492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514a4cf9-7434-43f9-8add-f1f884c14242-plugins-config-volume\") pod \"payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f\" (UID: \"514a4cf9-7434-43f9-8add-f1f884c14242\") " pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:34.336541 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.336523    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvf9f\" (UniqueName: \"kubernetes.io/projected/514a4cf9-7434-43f9-8add-f1f884c14242-kube-api-access-rvf9f\") pod \"payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f\" (UID: \"514a4cf9-7434-43f9-8add-f1f884c14242\") " pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:34.337160 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.337132    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4948d0f-71ec-46da-b5f9-df15a97ae835-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g\" (UID: \"c4948d0f-71ec-46da-b5f9-df15a97ae835\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:34.347615 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.347585    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl44v\" (UniqueName: \"kubernetes.io/projected/c4948d0f-71ec-46da-b5f9-df15a97ae835-kube-api-access-hl44v\") pod \"payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g\" (UID: \"c4948d0f-71ec-46da-b5f9-df15a97ae835\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:34.437243 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.437177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514a4cf9-7434-43f9-8add-f1f884c14242-plugins-config-volume\") pod \"payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f\" (UID: \"514a4cf9-7434-43f9-8add-f1f884c14242\") " pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:34.437243 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.437248    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvf9f\" (UniqueName: \"kubernetes.io/projected/514a4cf9-7434-43f9-8add-f1f884c14242-kube-api-access-rvf9f\") pod \"payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f\" (UID: \"514a4cf9-7434-43f9-8add-f1f884c14242\") " pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:34.437997 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.437968    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514a4cf9-7434-43f9-8add-f1f884c14242-plugins-config-volume\") pod \"payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f\" (UID: \"514a4cf9-7434-43f9-8add-f1f884c14242\") " pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:34.449715 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.449690    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvf9f\" (UniqueName: \"kubernetes.io/projected/514a4cf9-7434-43f9-8add-f1f884c14242-kube-api-access-rvf9f\") pod \"payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f\" (UID: \"514a4cf9-7434-43f9-8add-f1f884c14242\") " pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:34.518351 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.518310    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:34.625015 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.624976    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:34.696665 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.696589    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"]
Jul 29 01:04:34.696802 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:04:34.696774    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4948d0f_71ec_46da_b5f9_df15a97ae835.slice/crio-c4b00d3646142fb871b4dda9bb1a275f0048b97b98d1caf16abd79fbd2c4824d WatchSource:0}: Error finding container c4b00d3646142fb871b4dda9bb1a275f0048b97b98d1caf16abd79fbd2c4824d: Status 404 returned error can't find the container with id c4b00d3646142fb871b4dda9bb1a275f0048b97b98d1caf16abd79fbd2c4824d
Jul 29 01:04:34.814383 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:34.814354    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"]
Jul 29 01:04:34.815994 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:04:34.815919    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514a4cf9_7434_43f9_8add_f1f884c14242.slice/crio-3529cf253cf03e4aaa43d5c89116b3ac40a27882bfcd0c071bf44f1a19b775a1 WatchSource:0}: Error finding container 3529cf253cf03e4aaa43d5c89116b3ac40a27882bfcd0c071bf44f1a19b775a1: Status 404 returned error can't find the container with id 3529cf253cf03e4aaa43d5c89116b3ac40a27882bfcd0c071bf44f1a19b775a1
Jul 29 01:04:35.166158 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:35.166066    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f" event={"ID":"514a4cf9-7434-43f9-8add-f1f884c14242","Type":"ContainerStarted","Data":"c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4"}
Jul 29 01:04:35.166158 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:35.166128    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f" event={"ID":"514a4cf9-7434-43f9-8add-f1f884c14242","Type":"ContainerStarted","Data":"3529cf253cf03e4aaa43d5c89116b3ac40a27882bfcd0c071bf44f1a19b775a1"}
Jul 29 01:04:35.166419 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:35.166329    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:35.168087 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:35.168048    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g" event={"ID":"c4948d0f-71ec-46da-b5f9-df15a97ae835","Type":"ContainerStarted","Data":"afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2"}
Jul 29 01:04:35.168252 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:35.168104    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g" event={"ID":"c4948d0f-71ec-46da-b5f9-df15a97ae835","Type":"ContainerStarted","Data":"c4b00d3646142fb871b4dda9bb1a275f0048b97b98d1caf16abd79fbd2c4824d"}
Jul 29 01:04:35.168252 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:35.168141    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:35.185515 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:35.185450    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f" podStartSLOduration=1.1854254499999999 podStartE2EDuration="1.18542545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:04:35.185107224 +0000 UTC m=+1705.494035080" watchObservedRunningTime="2026-07-29 01:04:35.18542545 +0000 UTC m=+1705.494353299"
Jul 29 01:04:35.203827 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:35.203769    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g" podStartSLOduration=1.203754292 podStartE2EDuration="1.203754292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:04:35.202307466 +0000 UTC m=+1705.511235308" watchObservedRunningTime="2026-07-29 01:04:35.203754292 +0000 UTC m=+1705.512682134"
Jul 29 01:04:36.177486 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:36.177447    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz" event={"ID":"8f00ee30-2563-427d-b47a-61151e0d7fab","Type":"ContainerStarted","Data":"01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8"}
Jul 29 01:04:36.177949 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:36.177538    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:36.194725 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:36.194674    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz" podStartSLOduration=1.774254606 podStartE2EDuration="3.194659155s" totalImagesPullingTime="1.420404549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:04:36.19344885 +0000 UTC m=+1706.502376693" watchObservedRunningTime="2026-07-29 01:04:36.194659155 +0000 UTC m=+1706.503586997"
Jul 29 01:04:42.188601 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:42.188571    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:04:46.179825 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:46.179793    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:04:46.180223 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:46.179871    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:04:58.241904 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:58.241863    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"]
Jul 29 01:04:58.245426 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:58.245407    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"
Jul 29 01:04:58.248110 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:58.248086    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:04:58.253357 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:58.253330    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"]
Jul 29 01:04:58.362467 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:58.362426    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20982a6-5c70-4f29-b61d-f07adc68bcdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-956869dd-p469q\" (UID: \"b20982a6-5c70-4f29-b61d-f07adc68bcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"
Jul 29 01:04:58.462941 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:58.462900    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20982a6-5c70-4f29-b61d-f07adc68bcdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-956869dd-p469q\" (UID: \"b20982a6-5c70-4f29-b61d-f07adc68bcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"
Jul 29 01:04:58.463547 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:58.463529    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20982a6-5c70-4f29-b61d-f07adc68bcdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-956869dd-p469q\" (UID: \"b20982a6-5c70-4f29-b61d-f07adc68bcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"
Jul 29 01:04:58.558873 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:58.558767    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"
Jul 29 01:04:58.695914 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:58.695886    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"]
Jul 29 01:04:58.698693 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:04:58.698665    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20982a6_5c70_4f29_b61d_f07adc68bcdb.slice/crio-d1c61ec10b5d307efdefb236bc6977821010601e71701b32ad37c65be992db3f WatchSource:0}: Error finding container d1c61ec10b5d307efdefb236bc6977821010601e71701b32ad37c65be992db3f: Status 404 returned error can't find the container with id d1c61ec10b5d307efdefb236bc6977821010601e71701b32ad37c65be992db3f
Jul 29 01:04:59.318622 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:59.318582    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q" event={"ID":"b20982a6-5c70-4f29-b61d-f07adc68bcdb","Type":"ContainerStarted","Data":"a9a34d7f837a8c14ab380df78c1f1f80ee78da358ebaf82d4fa4c4df072a7a33"}
Jul 29 01:04:59.319069 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:59.318652    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q" event={"ID":"b20982a6-5c70-4f29-b61d-f07adc68bcdb","Type":"ContainerStarted","Data":"d1c61ec10b5d307efdefb236bc6977821010601e71701b32ad37c65be992db3f"}
Jul 29 01:04:59.335548 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:04:59.335490    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q" podStartSLOduration=1.335470659 podStartE2EDuration="1.335470659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:04:59.332684288 +0000 UTC m=+1729.641612132" watchObservedRunningTime="2026-07-29 01:04:59.335470659 +0000 UTC m=+1729.644398501"
Jul 29 01:05:00.325980 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:00.325935    2586 generic.go:356] "Generic (PLEG): container finished" podID="b20982a6-5c70-4f29-b61d-f07adc68bcdb" containerID="a9a34d7f837a8c14ab380df78c1f1f80ee78da358ebaf82d4fa4c4df072a7a33" exitCode=0
Jul 29 01:05:00.325980 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:00.325985    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q" event={"ID":"b20982a6-5c70-4f29-b61d-f07adc68bcdb","Type":"ContainerDied","Data":"a9a34d7f837a8c14ab380df78c1f1f80ee78da358ebaf82d4fa4c4df072a7a33"}
Jul 29 01:05:01.469466 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:01.469440    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"
Jul 29 01:05:01.591801 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:01.591715    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20982a6-5c70-4f29-b61d-f07adc68bcdb-maas-api-service-ca\") pod \"b20982a6-5c70-4f29-b61d-f07adc68bcdb\" (UID: \"b20982a6-5c70-4f29-b61d-f07adc68bcdb\") "
Jul 29 01:05:01.592084 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:01.592057    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20982a6-5c70-4f29-b61d-f07adc68bcdb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b20982a6-5c70-4f29-b61d-f07adc68bcdb" (UID: "b20982a6-5c70-4f29-b61d-f07adc68bcdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:05:01.692748 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:01.692693    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20982a6-5c70-4f29-b61d-f07adc68bcdb-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:02.338795 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:02.338755    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q" event={"ID":"b20982a6-5c70-4f29-b61d-f07adc68bcdb","Type":"ContainerDied","Data":"d1c61ec10b5d307efdefb236bc6977821010601e71701b32ad37c65be992db3f"}
Jul 29 01:05:02.338966 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:02.338808    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1c61ec10b5d307efdefb236bc6977821010601e71701b32ad37c65be992db3f"
Jul 29 01:05:02.338966 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:02.338766    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"
Jul 29 01:05:04.633347 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.633311    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"]
Jul 29 01:05:04.633811 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.633623    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz" podUID="8f00ee30-2563-427d-b47a-61151e0d7fab" containerName="maas-api" containerID="cri-o://01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8" gracePeriod=30
Jul 29 01:05:04.643491 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.643464    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:04.669576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.669540    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"]
Jul 29 01:05:04.669929 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.669847    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f" podUID="514a4cf9-7434-43f9-8add-f1f884c14242" containerName="payload-processing" containerID="cri-o://c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4" gracePeriod=30
Jul 29 01:05:04.678392 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.678361    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"]
Jul 29 01:05:04.679107 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.679031    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g" podUID="c4948d0f-71ec-46da-b5f9-df15a97ae835" containerName="payload-pre-processing" containerID="cri-o://afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2" gracePeriod=30
Jul 29 01:05:04.708311 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.708280    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:04.765811 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.765782    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:04.796612 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.796581    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"]
Jul 29 01:05:04.800512 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.800485    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-956869dd-p469q"]
Jul 29 01:05:04.838440 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.838401    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:04.945354 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.945155    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:04.953753 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.951267    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:05:04.978569 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.978534    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:05:04.986064 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:04.986044    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:05:05.128965 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.128930    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f00ee30-2563-427d-b47a-61151e0d7fab-openshift-service-ca\") pod \"8f00ee30-2563-427d-b47a-61151e0d7fab\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") "
Jul 29 01:05:05.129168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.128993    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5flt\" (UniqueName: \"kubernetes.io/projected/8f00ee30-2563-427d-b47a-61151e0d7fab-kube-api-access-w5flt\") pod \"8f00ee30-2563-427d-b47a-61151e0d7fab\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") "
Jul 29 01:05:05.129168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129031    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514a4cf9-7434-43f9-8add-f1f884c14242-plugins-config-volume\") pod \"514a4cf9-7434-43f9-8add-f1f884c14242\" (UID: \"514a4cf9-7434-43f9-8add-f1f884c14242\") "
Jul 29 01:05:05.129168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129075    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f00ee30-2563-427d-b47a-61151e0d7fab-maas-api-tls\") pod \"8f00ee30-2563-427d-b47a-61151e0d7fab\" (UID: \"8f00ee30-2563-427d-b47a-61151e0d7fab\") "
Jul 29 01:05:05.129168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129091    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl44v\" (UniqueName: \"kubernetes.io/projected/c4948d0f-71ec-46da-b5f9-df15a97ae835-kube-api-access-hl44v\") pod \"c4948d0f-71ec-46da-b5f9-df15a97ae835\" (UID: \"c4948d0f-71ec-46da-b5f9-df15a97ae835\") "
Jul 29 01:05:05.129168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129125    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4948d0f-71ec-46da-b5f9-df15a97ae835-plugins-config-volume\") pod \"c4948d0f-71ec-46da-b5f9-df15a97ae835\" (UID: \"c4948d0f-71ec-46da-b5f9-df15a97ae835\") "
Jul 29 01:05:05.129168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129146    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvf9f\" (UniqueName: \"kubernetes.io/projected/514a4cf9-7434-43f9-8add-f1f884c14242-kube-api-access-rvf9f\") pod \"514a4cf9-7434-43f9-8add-f1f884c14242\" (UID: \"514a4cf9-7434-43f9-8add-f1f884c14242\") "
Jul 29 01:05:05.129516 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129459    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f00ee30-2563-427d-b47a-61151e0d7fab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f00ee30-2563-427d-b47a-61151e0d7fab" (UID: "8f00ee30-2563-427d-b47a-61151e0d7fab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:05:05.129563 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129506    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514a4cf9-7434-43f9-8add-f1f884c14242-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "514a4cf9-7434-43f9-8add-f1f884c14242" (UID: "514a4cf9-7434-43f9-8add-f1f884c14242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:05:05.129677 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129634    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4948d0f-71ec-46da-b5f9-df15a97ae835-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4948d0f-71ec-46da-b5f9-df15a97ae835" (UID: "c4948d0f-71ec-46da-b5f9-df15a97ae835"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:05:05.129916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129843    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f00ee30-2563-427d-b47a-61151e0d7fab-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:05.129916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129869    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514a4cf9-7434-43f9-8add-f1f884c14242-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:05.129916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.129885    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4948d0f-71ec-46da-b5f9-df15a97ae835-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:05.131800 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.131773    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f00ee30-2563-427d-b47a-61151e0d7fab-kube-api-access-w5flt" (OuterVolumeSpecName: "kube-api-access-w5flt") pod "8f00ee30-2563-427d-b47a-61151e0d7fab" (UID: "8f00ee30-2563-427d-b47a-61151e0d7fab"). InnerVolumeSpecName "kube-api-access-w5flt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:05:05.132483 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.132446    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4948d0f-71ec-46da-b5f9-df15a97ae835-kube-api-access-hl44v" (OuterVolumeSpecName: "kube-api-access-hl44v") pod "c4948d0f-71ec-46da-b5f9-df15a97ae835" (UID: "c4948d0f-71ec-46da-b5f9-df15a97ae835"). InnerVolumeSpecName "kube-api-access-hl44v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:05:05.132565 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.132470    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514a4cf9-7434-43f9-8add-f1f884c14242-kube-api-access-rvf9f" (OuterVolumeSpecName: "kube-api-access-rvf9f") pod "514a4cf9-7434-43f9-8add-f1f884c14242" (UID: "514a4cf9-7434-43f9-8add-f1f884c14242"). InnerVolumeSpecName "kube-api-access-rvf9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:05:05.132565 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.132539    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f00ee30-2563-427d-b47a-61151e0d7fab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f00ee30-2563-427d-b47a-61151e0d7fab" (UID: "8f00ee30-2563-427d-b47a-61151e0d7fab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:05:05.173399 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.173336    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:05.230692 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.230662    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5flt\" (UniqueName: \"kubernetes.io/projected/8f00ee30-2563-427d-b47a-61151e0d7fab-kube-api-access-w5flt\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:05.230692 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.230688    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f00ee30-2563-427d-b47a-61151e0d7fab-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:05.230885 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.230703    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl44v\" (UniqueName: \"kubernetes.io/projected/c4948d0f-71ec-46da-b5f9-df15a97ae835-kube-api-access-hl44v\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:05.230885 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.230712    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvf9f\" (UniqueName: \"kubernetes.io/projected/514a4cf9-7434-43f9-8add-f1f884c14242-kube-api-access-rvf9f\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:05.357560 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.357517    2586 generic.go:356] "Generic (PLEG): container finished" podID="8f00ee30-2563-427d-b47a-61151e0d7fab" containerID="01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8" exitCode=0
Jul 29 01:05:05.357752 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.357569    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"
Jul 29 01:05:05.357752 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.357601    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz" event={"ID":"8f00ee30-2563-427d-b47a-61151e0d7fab","Type":"ContainerDied","Data":"01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8"}
Jul 29 01:05:05.357752 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.357655    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz" event={"ID":"8f00ee30-2563-427d-b47a-61151e0d7fab","Type":"ContainerDied","Data":"cb8c7c3e0cdb4665ef16b96d9b5e8d3f40a66d0eb3e1666fc1d0a9c500d1c1cf"}
Jul 29 01:05:05.357752 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.357687    2586 scope.go:117] "RemoveContainer" containerID="01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8"
Jul 29 01:05:05.359099 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.359072    2586 generic.go:356] "Generic (PLEG): container finished" podID="514a4cf9-7434-43f9-8add-f1f884c14242" containerID="c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4" exitCode=0
Jul 29 01:05:05.359232 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.359140    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"
Jul 29 01:05:05.359232 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.359158    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f" event={"ID":"514a4cf9-7434-43f9-8add-f1f884c14242","Type":"ContainerDied","Data":"c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4"}
Jul 29 01:05:05.359370 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.359245    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f" event={"ID":"514a4cf9-7434-43f9-8add-f1f884c14242","Type":"ContainerDied","Data":"3529cf253cf03e4aaa43d5c89116b3ac40a27882bfcd0c071bf44f1a19b775a1"}
Jul 29 01:05:05.360832 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.360808    2586 generic.go:356] "Generic (PLEG): container finished" podID="c4948d0f-71ec-46da-b5f9-df15a97ae835" containerID="afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2" exitCode=0
Jul 29 01:05:05.360969 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.360871    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g" event={"ID":"c4948d0f-71ec-46da-b5f9-df15a97ae835","Type":"ContainerDied","Data":"afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2"}
Jul 29 01:05:05.360969 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.360900    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g" event={"ID":"c4948d0f-71ec-46da-b5f9-df15a97ae835","Type":"ContainerDied","Data":"c4b00d3646142fb871b4dda9bb1a275f0048b97b98d1caf16abd79fbd2c4824d"}
Jul 29 01:05:05.360969 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.360961    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"
Jul 29 01:05:05.369458 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.369082    2586 scope.go:117] "RemoveContainer" containerID="01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8"
Jul 29 01:05:05.369958 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:05.369833    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8\": container with ID starting with 01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8 not found: ID does not exist" containerID="01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8"
Jul 29 01:05:05.369958 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.369874    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8"} err="failed to get container status \"01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8\": rpc error: code = NotFound desc = could not find container \"01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8\": container with ID starting with 01d082fda9c2aa54f4f779888aa3b67794d087011ee17d138cf34bb842e67bf8 not found: ID does not exist"
Jul 29 01:05:05.369958 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.369898    2586 scope.go:117] "RemoveContainer" containerID="c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4"
Jul 29 01:05:05.383753 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.383732    2586 scope.go:117] "RemoveContainer" containerID="c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4"
Jul 29 01:05:05.384058 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:05.384037    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4\": container with ID starting with c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4 not found: ID does not exist" containerID="c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4"
Jul 29 01:05:05.384112 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.384067    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4"} err="failed to get container status \"c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4\": rpc error: code = NotFound desc = could not find container \"c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4\": container with ID starting with c98251adcf08aa19ef068469a0893904e5fc0c49dda7d0d6e6b6545368d8b8f4 not found: ID does not exist"
Jul 29 01:05:05.384112 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.384085    2586 scope.go:117] "RemoveContainer" containerID="afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2"
Jul 29 01:05:05.385901 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.385879    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"]
Jul 29 01:05:05.390379 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.390356    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-956869dd-55b87f4ffb-jw9xz"]
Jul 29 01:05:05.392311 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.392282    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:05.393615 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.393599    2586 scope.go:117] "RemoveContainer" containerID="afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2"
Jul 29 01:05:05.393856 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:05.393838    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2\": container with ID starting with afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2 not found: ID does not exist" containerID="afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2"
Jul 29 01:05:05.393920 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.393866    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2"} err="failed to get container status \"afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2\": rpc error: code = NotFound desc = could not find container \"afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2\": container with ID starting with afcb8daf9b7c0b993c01344c8e29a1d7e00f4e7c198078dd6de85cf93c99d3b2 not found: ID does not exist"
Jul 29 01:05:05.400482 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.400461    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"]
Jul 29 01:05:05.404727 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.404708    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-956869dd-5958694f9b-dj58g"]
Jul 29 01:05:05.417921 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.417898    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"]
Jul 29 01:05:05.422066 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.422046    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-956869dd-5cfb487dbf-wlc8f"]
Jul 29 01:05:05.698382 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:05.698346    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:06.048031 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:06.047951    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:06.221980 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:06.221943    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="514a4cf9-7434-43f9-8add-f1f884c14242" path="/var/lib/kubelet/pods/514a4cf9-7434-43f9-8add-f1f884c14242/volumes"
Jul 29 01:05:06.222378 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:06.222364    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f00ee30-2563-427d-b47a-61151e0d7fab" path="/var/lib/kubelet/pods/8f00ee30-2563-427d-b47a-61151e0d7fab/volumes"
Jul 29 01:05:06.222668 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:06.222656    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b20982a6-5c70-4f29-b61d-f07adc68bcdb" path="/var/lib/kubelet/pods/b20982a6-5c70-4f29-b61d-f07adc68bcdb/volumes"
Jul 29 01:05:06.222965 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:06.222954    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4948d0f-71ec-46da-b5f9-df15a97ae835" path="/var/lib/kubelet/pods/c4948d0f-71ec-46da-b5f9-df15a97ae835/volumes"
Jul 29 01:05:08.332412 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.332374    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:08.418575 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.418535    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:08.419960 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.419934    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"]
Jul 29 01:05:08.420390 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420372    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514a4cf9-7434-43f9-8add-f1f884c14242" containerName="payload-processing"
Jul 29 01:05:08.420481 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420395    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="514a4cf9-7434-43f9-8add-f1f884c14242" containerName="payload-processing"
Jul 29 01:05:08.420481 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420424    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4948d0f-71ec-46da-b5f9-df15a97ae835" containerName="payload-pre-processing"
Jul 29 01:05:08.420481 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420433    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4948d0f-71ec-46da-b5f9-df15a97ae835" containerName="payload-pre-processing"
Jul 29 01:05:08.420481 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420466    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f00ee30-2563-427d-b47a-61151e0d7fab" containerName="maas-api"
Jul 29 01:05:08.420481 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420475    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f00ee30-2563-427d-b47a-61151e0d7fab" containerName="maas-api"
Jul 29 01:05:08.420736 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420489    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b20982a6-5c70-4f29-b61d-f07adc68bcdb" containerName="revoke-keys"
Jul 29 01:05:08.420736 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420499    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b20982a6-5c70-4f29-b61d-f07adc68bcdb" containerName="revoke-keys"
Jul 29 01:05:08.420736 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420586    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4948d0f-71ec-46da-b5f9-df15a97ae835" containerName="payload-pre-processing"
Jul 29 01:05:08.420736 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420602    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f00ee30-2563-427d-b47a-61151e0d7fab" containerName="maas-api"
Jul 29 01:05:08.420736 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420613    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="514a4cf9-7434-43f9-8add-f1f884c14242" containerName="payload-processing"
Jul 29 01:05:08.420736 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.420623    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b20982a6-5c70-4f29-b61d-f07adc68bcdb" containerName="revoke-keys"
Jul 29 01:05:08.427009 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.426986    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:08.430868 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.430834    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c63818d3-serving-cert\""
Jul 29 01:05:08.431008 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.430892    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 01:05:08.431008 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.430840    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9jxwk\""
Jul 29 01:05:08.431120 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.431108    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:05:08.434177 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.434008    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"]
Jul 29 01:05:08.523921 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.523888    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:08.563356 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.563321    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7eb661c-1a31-4389-a827-b69af585e0e2-maas-api-tls\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:08.563356 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.563357    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb661c-1a31-4389-a827-b69af585e0e2-openshift-service-ca\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:08.563635 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.563468    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzj4\" (UniqueName: \"kubernetes.io/projected/a7eb661c-1a31-4389-a827-b69af585e0e2-kube-api-access-pwzj4\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:08.665056 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.664953    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7eb661c-1a31-4389-a827-b69af585e0e2-maas-api-tls\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:08.665056 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.665013    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb661c-1a31-4389-a827-b69af585e0e2-openshift-service-ca\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:08.665316 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:08.665106    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-serving-cert: secret "maas-api-e2e-ait-c63818d3-serving-cert" not found
Jul 29 01:05:08.665316 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.665111    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzj4\" (UniqueName: \"kubernetes.io/projected/a7eb661c-1a31-4389-a827-b69af585e0e2-kube-api-access-pwzj4\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:08.665316 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:08.665176    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7eb661c-1a31-4389-a827-b69af585e0e2-maas-api-tls podName:a7eb661c-1a31-4389-a827-b69af585e0e2 nodeName:}" failed. No retries permitted until 2026-07-29 01:05:09.165159254 +0000 UTC m=+1739.474087074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7eb661c-1a31-4389-a827-b69af585e0e2-maas-api-tls") pod "maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz" (UID: "a7eb661c-1a31-4389-a827-b69af585e0e2") : secret "maas-api-e2e-ait-c63818d3-serving-cert" not found
Jul 29 01:05:08.665715 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.665695    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb661c-1a31-4389-a827-b69af585e0e2-openshift-service-ca\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:08.673526 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.673503    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzj4\" (UniqueName: \"kubernetes.io/projected/a7eb661c-1a31-4389-a827-b69af585e0e2-kube-api-access-pwzj4\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:08.716703 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.716672    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:08.969881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:08.969785    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:09.071953 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.071915    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:09.072785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.072763    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"]
Jul 29 01:05:09.077603 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.077576    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:09.080385 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.080345    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c63818d3\""
Jul 29 01:05:09.080385 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.080364    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c63818d3-dockercfg-x6p2c\""
Jul 29 01:05:09.087383 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.087356    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"]
Jul 29 01:05:09.116593 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.116560    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"]
Jul 29 01:05:09.120954 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.120929    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:09.130138 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.130106    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"]
Jul 29 01:05:09.171322 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.171288    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7eb661c-1a31-4389-a827-b69af585e0e2-maas-api-tls\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:09.174092 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.174067    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7eb661c-1a31-4389-a827-b69af585e0e2-maas-api-tls\") pod \"maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:09.272499 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.272395    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd\" (UID: \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\") " pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:09.272499 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.272473    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpkp\" (UniqueName: \"kubernetes.io/projected/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-kube-api-access-4bpkp\") pod \"payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4\" (UID: \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:09.272695 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.272602    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4\" (UID: \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:09.272733 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.272690    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9r7w\" (UniqueName: \"kubernetes.io/projected/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-kube-api-access-j9r7w\") pod \"payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd\" (UID: \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\") " pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:09.341403 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.341359    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:09.373960 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.373920    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4\" (UID: \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:09.374144 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.374002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9r7w\" (UniqueName: \"kubernetes.io/projected/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-kube-api-access-j9r7w\") pod \"payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd\" (UID: \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\") " pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:09.374144 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.374036    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd\" (UID: \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\") " pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:09.374144 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.374093    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpkp\" (UniqueName: \"kubernetes.io/projected/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-kube-api-access-4bpkp\") pod \"payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4\" (UID: \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:09.374805 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.374777    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd\" (UID: \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\") " pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:09.375240 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.375212    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4\" (UID: \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:09.383031 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.382939    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpkp\" (UniqueName: \"kubernetes.io/projected/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-kube-api-access-4bpkp\") pod \"payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4\" (UID: \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:09.383031 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.382999    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9r7w\" (UniqueName: \"kubernetes.io/projected/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-kube-api-access-j9r7w\") pod \"payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd\" (UID: \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\") " pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:09.392161 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.392127    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:09.434042 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.433779    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:09.529261 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.529072    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"]
Jul 29 01:05:09.530649 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:05:09.530602    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7eb661c_1a31_4389_a827_b69af585e0e2.slice/crio-4c8aaef0ee8226b69fa970c565d833cd7fb65c6c3b186add726748edb2303fe2 WatchSource:0}: Error finding container 4c8aaef0ee8226b69fa970c565d833cd7fb65c6c3b186add726748edb2303fe2: Status 404 returned error can't find the container with id 4c8aaef0ee8226b69fa970c565d833cd7fb65c6c3b186add726748edb2303fe2
Jul 29 01:05:09.532787 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.532763    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:05:09.583923 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.583863    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"]
Jul 29 01:05:09.586811 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:05:09.586770    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d84d2d_8a7c_4c4f_b32f_cd1bee7657b0.slice/crio-d8bd8e4fd5d07014060e83de83a98f7e62391f13386eae81a29b03d4b3a7ad34 WatchSource:0}: Error finding container d8bd8e4fd5d07014060e83de83a98f7e62391f13386eae81a29b03d4b3a7ad34: Status 404 returned error can't find the container with id d8bd8e4fd5d07014060e83de83a98f7e62391f13386eae81a29b03d4b3a7ad34
Jul 29 01:05:09.613938 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:09.613911    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"]
Jul 29 01:05:09.621414 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:05:09.621380    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb385fc3_1d13_49f8_8fc9_fbdbb19c31e2.slice/crio-0ff32ff7d5e7169a41f4f78f1d985404b83f7f2801bb329d365916d19f9dba2b WatchSource:0}: Error finding container 0ff32ff7d5e7169a41f4f78f1d985404b83f7f2801bb329d365916d19f9dba2b: Status 404 returned error can't find the container with id 0ff32ff7d5e7169a41f4f78f1d985404b83f7f2801bb329d365916d19f9dba2b
Jul 29 01:05:10.397291 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.397251    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz" event={"ID":"a7eb661c-1a31-4389-a827-b69af585e0e2","Type":"ContainerStarted","Data":"d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300"}
Jul 29 01:05:10.397774 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.397310    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz" event={"ID":"a7eb661c-1a31-4389-a827-b69af585e0e2","Type":"ContainerStarted","Data":"4c8aaef0ee8226b69fa970c565d833cd7fb65c6c3b186add726748edb2303fe2"}
Jul 29 01:05:10.397774 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.397337    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:10.399168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.399141    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd" event={"ID":"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2","Type":"ContainerStarted","Data":"5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6"}
Jul 29 01:05:10.399325 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.399187    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd" event={"ID":"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2","Type":"ContainerStarted","Data":"0ff32ff7d5e7169a41f4f78f1d985404b83f7f2801bb329d365916d19f9dba2b"}
Jul 29 01:05:10.399325 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.399267    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:10.400799 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.400774    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4" event={"ID":"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0","Type":"ContainerStarted","Data":"1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88"}
Jul 29 01:05:10.400949 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.400815    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4" event={"ID":"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0","Type":"ContainerStarted","Data":"d8bd8e4fd5d07014060e83de83a98f7e62391f13386eae81a29b03d4b3a7ad34"}
Jul 29 01:05:10.400949 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.400879    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:10.416114 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.415981    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz" podStartSLOduration=2.031146193 podStartE2EDuration="2.415962215s" totalImagesPullingTime="384.816022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:05:10.415909836 +0000 UTC m=+1740.724837680" watchObservedRunningTime="2026-07-29 01:05:10.415962215 +0000 UTC m=+1740.724890062"
Jul 29 01:05:10.433472 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.433410    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4" podStartSLOduration=1.433385693 podStartE2EDuration="1.433385693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:05:10.431737441 +0000 UTC m=+1740.740665285" watchObservedRunningTime="2026-07-29 01:05:10.433385693 +0000 UTC m=+1740.742313538"
Jul 29 01:05:10.447620 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:10.447562    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd" podStartSLOduration=1.447548654 podStartE2EDuration="1.447548654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:05:10.446113466 +0000 UTC m=+1740.755041309" watchObservedRunningTime="2026-07-29 01:05:10.447548654 +0000 UTC m=+1740.756476496"
Jul 29 01:05:16.412294 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:16.412261    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:21.408723 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:21.408688    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:21.409142 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:21.408761    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:34.042103 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.042067    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"]
Jul 29 01:05:34.045789 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.045768    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"
Jul 29 01:05:34.048555 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.048486    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:05:34.051992 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.051961    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"]
Jul 29 01:05:34.205728 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.205685    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/605543b6-d045-4514-b741-eefed280d0d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c63818d3-qs76m\" (UID: \"605543b6-d045-4514-b741-eefed280d0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"
Jul 29 01:05:34.306434 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.306341    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/605543b6-d045-4514-b741-eefed280d0d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c63818d3-qs76m\" (UID: \"605543b6-d045-4514-b741-eefed280d0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"
Jul 29 01:05:34.306996 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.306976    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/605543b6-d045-4514-b741-eefed280d0d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c63818d3-qs76m\" (UID: \"605543b6-d045-4514-b741-eefed280d0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"
Jul 29 01:05:34.358975 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.358939    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"
Jul 29 01:05:34.504643 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.504611    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"]
Jul 29 01:05:34.506417 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:05:34.506388    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605543b6_d045_4514_b741_eefed280d0d4.slice/crio-c87232ed43f86f86263197199cc1a879c49059cc6963ec15ebe4dcb17b7d2340 WatchSource:0}: Error finding container c87232ed43f86f86263197199cc1a879c49059cc6963ec15ebe4dcb17b7d2340: Status 404 returned error can't find the container with id c87232ed43f86f86263197199cc1a879c49059cc6963ec15ebe4dcb17b7d2340
Jul 29 01:05:34.542605 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:34.542572    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m" event={"ID":"605543b6-d045-4514-b741-eefed280d0d4","Type":"ContainerStarted","Data":"c87232ed43f86f86263197199cc1a879c49059cc6963ec15ebe4dcb17b7d2340"}
Jul 29 01:05:35.549804 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:35.549708    2586 generic.go:356] "Generic (PLEG): container finished" podID="605543b6-d045-4514-b741-eefed280d0d4" containerID="3c54476625b0de489a0168f866d11f3c5a96ad17dce1059e711a19be16aefffd" exitCode=0
Jul 29 01:05:35.549804 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:35.549756    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m" event={"ID":"605543b6-d045-4514-b741-eefed280d0d4","Type":"ContainerDied","Data":"3c54476625b0de489a0168f866d11f3c5a96ad17dce1059e711a19be16aefffd"}
Jul 29 01:05:36.698018 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:36.697988    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"
Jul 29 01:05:36.830604 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:36.830503    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/605543b6-d045-4514-b741-eefed280d0d4-maas-api-service-ca\") pod \"605543b6-d045-4514-b741-eefed280d0d4\" (UID: \"605543b6-d045-4514-b741-eefed280d0d4\") "
Jul 29 01:05:36.830892 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:36.830867    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605543b6-d045-4514-b741-eefed280d0d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "605543b6-d045-4514-b741-eefed280d0d4" (UID: "605543b6-d045-4514-b741-eefed280d0d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:05:36.931536 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:36.931495    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/605543b6-d045-4514-b741-eefed280d0d4-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:37.564278 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:37.564248    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"
Jul 29 01:05:37.564278 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:37.564262    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m" event={"ID":"605543b6-d045-4514-b741-eefed280d0d4","Type":"ContainerDied","Data":"c87232ed43f86f86263197199cc1a879c49059cc6963ec15ebe4dcb17b7d2340"}
Jul 29 01:05:37.564513 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:37.564318    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87232ed43f86f86263197199cc1a879c49059cc6963ec15ebe4dcb17b7d2340"
Jul 29 01:05:38.904564 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:38.904528    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"]
Jul 29 01:05:38.904987 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:38.904771    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz" podUID="a7eb661c-1a31-4389-a827-b69af585e0e2" containerName="maas-api" containerID="cri-o://d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300" gracePeriod=30
Jul 29 01:05:38.930416 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:38.930393    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:38.958311 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:38.958276    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"]
Jul 29 01:05:38.958542 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:38.958515    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd" podUID="db385fc3-1d13-49f8-8fc9-fbdbb19c31e2" containerName="payload-processing" containerID="cri-o://5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6" gracePeriod=30
Jul 29 01:05:38.967083 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:38.967051    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"]
Jul 29 01:05:38.967339 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:38.967314    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4" podUID="90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0" containerName="payload-pre-processing" containerID="cri-o://1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88" gracePeriod=30
Jul 29 01:05:39.003841 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.003706    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:39.068087 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.068055    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:39.099748 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.099708    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"]
Jul 29 01:05:39.103239 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.103207    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c63818d3-qs76m"]
Jul 29 01:05:39.121809 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.121737    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:39.252401 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.252376    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:39.278875 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.278850    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:39.282044 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.282024    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:39.353094 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.353063    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwzj4\" (UniqueName: \"kubernetes.io/projected/a7eb661c-1a31-4389-a827-b69af585e0e2-kube-api-access-pwzj4\") pod \"a7eb661c-1a31-4389-a827-b69af585e0e2\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") "
Jul 29 01:05:39.353283 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.353126    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-plugins-config-volume\") pod \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\" (UID: \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\") "
Jul 29 01:05:39.353283 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.353175    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb661c-1a31-4389-a827-b69af585e0e2-openshift-service-ca\") pod \"a7eb661c-1a31-4389-a827-b69af585e0e2\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") "
Jul 29 01:05:39.353283 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.353224    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7eb661c-1a31-4389-a827-b69af585e0e2-maas-api-tls\") pod \"a7eb661c-1a31-4389-a827-b69af585e0e2\" (UID: \"a7eb661c-1a31-4389-a827-b69af585e0e2\") "
Jul 29 01:05:39.353283 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.353278    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9r7w\" (UniqueName: \"kubernetes.io/projected/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-kube-api-access-j9r7w\") pod \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\" (UID: \"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2\") "
Jul 29 01:05:39.354039 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.353584    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7eb661c-1a31-4389-a827-b69af585e0e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a7eb661c-1a31-4389-a827-b69af585e0e2" (UID: "a7eb661c-1a31-4389-a827-b69af585e0e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:05:39.354039 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.353689    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb661c-1a31-4389-a827-b69af585e0e2-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:39.354039 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.353801    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db385fc3-1d13-49f8-8fc9-fbdbb19c31e2" (UID: "db385fc3-1d13-49f8-8fc9-fbdbb19c31e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:05:39.355566 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.355536    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7eb661c-1a31-4389-a827-b69af585e0e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a7eb661c-1a31-4389-a827-b69af585e0e2" (UID: "a7eb661c-1a31-4389-a827-b69af585e0e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:05:39.355879 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.355858    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-kube-api-access-j9r7w" (OuterVolumeSpecName: "kube-api-access-j9r7w") pod "db385fc3-1d13-49f8-8fc9-fbdbb19c31e2" (UID: "db385fc3-1d13-49f8-8fc9-fbdbb19c31e2"). InnerVolumeSpecName "kube-api-access-j9r7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:05:39.355957 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.355888    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7eb661c-1a31-4389-a827-b69af585e0e2-kube-api-access-pwzj4" (OuterVolumeSpecName: "kube-api-access-pwzj4") pod "a7eb661c-1a31-4389-a827-b69af585e0e2" (UID: "a7eb661c-1a31-4389-a827-b69af585e0e2"). InnerVolumeSpecName "kube-api-access-pwzj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:05:39.454712 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.454620    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-plugins-config-volume\") pod \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\" (UID: \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\") "
Jul 29 01:05:39.454870 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.454715    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bpkp\" (UniqueName: \"kubernetes.io/projected/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-kube-api-access-4bpkp\") pod \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\" (UID: \"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0\") "
Jul 29 01:05:39.454978 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.454958    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:39.455049 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.454983    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7eb661c-1a31-4389-a827-b69af585e0e2-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:39.455049 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.454998    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9r7w\" (UniqueName: \"kubernetes.io/projected/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2-kube-api-access-j9r7w\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:39.455049 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.455013    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwzj4\" (UniqueName: \"kubernetes.io/projected/a7eb661c-1a31-4389-a827-b69af585e0e2-kube-api-access-pwzj4\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:39.455161 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.455040    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0" (UID: "90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:05:39.456793 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.456770    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-kube-api-access-4bpkp" (OuterVolumeSpecName: "kube-api-access-4bpkp") pod "90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0" (UID: "90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0"). InnerVolumeSpecName "kube-api-access-4bpkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:05:39.556114 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.556064    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:39.556114 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.556108    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bpkp\" (UniqueName: \"kubernetes.io/projected/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0-kube-api-access-4bpkp\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:05:39.577101 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.577065    2586 generic.go:356] "Generic (PLEG): container finished" podID="a7eb661c-1a31-4389-a827-b69af585e0e2" containerID="d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300" exitCode=0
Jul 29 01:05:39.577341 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.577117    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"
Jul 29 01:05:39.577341 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.577151    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz" event={"ID":"a7eb661c-1a31-4389-a827-b69af585e0e2","Type":"ContainerDied","Data":"d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300"}
Jul 29 01:05:39.577341 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.577231    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz" event={"ID":"a7eb661c-1a31-4389-a827-b69af585e0e2","Type":"ContainerDied","Data":"4c8aaef0ee8226b69fa970c565d833cd7fb65c6c3b186add726748edb2303fe2"}
Jul 29 01:05:39.577341 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.577263    2586 scope.go:117] "RemoveContainer" containerID="d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300"
Jul 29 01:05:39.578527 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.578497    2586 generic.go:356] "Generic (PLEG): container finished" podID="db385fc3-1d13-49f8-8fc9-fbdbb19c31e2" containerID="5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6" exitCode=0
Jul 29 01:05:39.578632 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.578545    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"
Jul 29 01:05:39.578632 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.578587    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd" event={"ID":"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2","Type":"ContainerDied","Data":"5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6"}
Jul 29 01:05:39.578754 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.578642    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd" event={"ID":"db385fc3-1d13-49f8-8fc9-fbdbb19c31e2","Type":"ContainerDied","Data":"0ff32ff7d5e7169a41f4f78f1d985404b83f7f2801bb329d365916d19f9dba2b"}
Jul 29 01:05:39.580126 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.580005    2586 generic.go:356] "Generic (PLEG): container finished" podID="90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0" containerID="1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88" exitCode=0
Jul 29 01:05:39.580126 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.580051    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4" event={"ID":"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0","Type":"ContainerDied","Data":"1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88"}
Jul 29 01:05:39.580126 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.580065    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"
Jul 29 01:05:39.580126 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.580081    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4" event={"ID":"90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0","Type":"ContainerDied","Data":"d8bd8e4fd5d07014060e83de83a98f7e62391f13386eae81a29b03d4b3a7ad34"}
Jul 29 01:05:39.588769 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.588748    2586 scope.go:117] "RemoveContainer" containerID="d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300"
Jul 29 01:05:39.589040 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:39.589022    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300\": container with ID starting with d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300 not found: ID does not exist" containerID="d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300"
Jul 29 01:05:39.589089 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.589049    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300"} err="failed to get container status \"d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300\": rpc error: code = NotFound desc = could not find container \"d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300\": container with ID starting with d5af783ae82c74bc800c9d3374a35f671272b06667ec0c13a626b59966f16300 not found: ID does not exist"
Jul 29 01:05:39.589089 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.589067    2586 scope.go:117] "RemoveContainer" containerID="5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6"
Jul 29 01:05:39.598063 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.598043    2586 scope.go:117] "RemoveContainer" containerID="5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6"
Jul 29 01:05:39.598355 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:39.598336    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6\": container with ID starting with 5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6 not found: ID does not exist" containerID="5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6"
Jul 29 01:05:39.598433 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.598360    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6"} err="failed to get container status \"5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6\": rpc error: code = NotFound desc = could not find container \"5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6\": container with ID starting with 5861b7cbb981248586e995bbfdded1f86300757173363c800cb052d912472fe6 not found: ID does not exist"
Jul 29 01:05:39.598433 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.598377    2586 scope.go:117] "RemoveContainer" containerID="1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88"
Jul 29 01:05:39.605764 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.605743    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"]
Jul 29 01:05:39.606642 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.606623    2586 scope.go:117] "RemoveContainer" containerID="1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88"
Jul 29 01:05:39.606892 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:39.606876    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88\": container with ID starting with 1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88 not found: ID does not exist" containerID="1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88"
Jul 29 01:05:39.606939 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.606898    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88"} err="failed to get container status \"1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88\": rpc error: code = NotFound desc = could not find container \"1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88\": container with ID starting with 1bffd1908d66c66c3c637d0e31d8f8c998462114fb71f9f4cc6e2e7fea944a88 not found: ID does not exist"
Jul 29 01:05:39.608005 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.607987    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c63818d3-c747d6b4-z5xcz"]
Jul 29 01:05:39.620844 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.620823    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"]
Jul 29 01:05:39.624411 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.624391    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c63818d3-55cdf47464-76tzd"]
Jul 29 01:05:39.635258 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.635237    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"]
Jul 29 01:05:39.639289 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:39.639268    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c63818d3-57d696fd5b-4m5w4"]
Jul 29 01:05:40.208402 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:40.208317    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:40.222384 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:40.222353    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="605543b6-d045-4514-b741-eefed280d0d4" path="/var/lib/kubelet/pods/605543b6-d045-4514-b741-eefed280d0d4/volumes"
Jul 29 01:05:40.222816 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:40.222796    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0" path="/var/lib/kubelet/pods/90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0/volumes"
Jul 29 01:05:40.223289 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:40.223270    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7eb661c-1a31-4389-a827-b69af585e0e2" path="/var/lib/kubelet/pods/a7eb661c-1a31-4389-a827-b69af585e0e2/volumes"
Jul 29 01:05:40.223682 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:40.223652    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db385fc3-1d13-49f8-8fc9-fbdbb19c31e2" path="/var/lib/kubelet/pods/db385fc3-1d13-49f8-8fc9-fbdbb19c31e2/volumes"
Jul 29 01:05:40.422521 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:40.422485    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:40.708801 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:40.708736    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:40.942412 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:40.942365    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:41.253528 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:41.253493    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:43.552139 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.552103    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:43.615653 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.615618    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"]
Jul 29 01:05:43.616170 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616154    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605543b6-d045-4514-b741-eefed280d0d4" containerName="revoke-keys"
Jul 29 01:05:43.616239 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616174    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="605543b6-d045-4514-b741-eefed280d0d4" containerName="revoke-keys"
Jul 29 01:05:43.616239 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616214    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7eb661c-1a31-4389-a827-b69af585e0e2" containerName="maas-api"
Jul 29 01:05:43.616239 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616223    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7eb661c-1a31-4389-a827-b69af585e0e2" containerName="maas-api"
Jul 29 01:05:43.616239 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616234    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db385fc3-1d13-49f8-8fc9-fbdbb19c31e2" containerName="payload-processing"
Jul 29 01:05:43.616371 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616242    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="db385fc3-1d13-49f8-8fc9-fbdbb19c31e2" containerName="payload-processing"
Jul 29 01:05:43.616371 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616254    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0" containerName="payload-pre-processing"
Jul 29 01:05:43.616371 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616262    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0" containerName="payload-pre-processing"
Jul 29 01:05:43.616371 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616365    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7eb661c-1a31-4389-a827-b69af585e0e2" containerName="maas-api"
Jul 29 01:05:43.616493 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616378    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d84d2d-8a7c-4c4f-b32f-cd1bee7657b0" containerName="payload-pre-processing"
Jul 29 01:05:43.616493 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616387    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="db385fc3-1d13-49f8-8fc9-fbdbb19c31e2" containerName="payload-processing"
Jul 29 01:05:43.616493 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.616397    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="605543b6-d045-4514-b741-eefed280d0d4" containerName="revoke-keys"
Jul 29 01:05:43.619788 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.619763    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:43.622369 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.622328    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-030a9f65-serving-cert\""
Jul 29 01:05:43.622369 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.622358    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 01:05:43.622605 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.622433    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:05:43.623420 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.623397    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9jxwk\""
Jul 29 01:05:43.630502 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.630473    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"]
Jul 29 01:05:43.637354 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.637325    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:43.703085 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.703057    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:43.795630 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.795592    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnt2s\" (UniqueName: \"kubernetes.io/projected/3e920794-4657-4c34-80dd-dea0986491a0-kube-api-access-gnt2s\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:43.795785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.795676    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e920794-4657-4c34-80dd-dea0986491a0-openshift-service-ca\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:43.795785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.795730    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e920794-4657-4c34-80dd-dea0986491a0-maas-api-tls\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:43.896973 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.896931    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e920794-4657-4c34-80dd-dea0986491a0-maas-api-tls\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:43.897153 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.896995    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnt2s\" (UniqueName: \"kubernetes.io/projected/3e920794-4657-4c34-80dd-dea0986491a0-kube-api-access-gnt2s\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:43.897153 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.897039    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e920794-4657-4c34-80dd-dea0986491a0-openshift-service-ca\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:43.897153 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:43.897091    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-serving-cert: secret "maas-api-e2e-derive-030a9f65-serving-cert" not found
Jul 29 01:05:43.897337 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:05:43.897180    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e920794-4657-4c34-80dd-dea0986491a0-maas-api-tls podName:3e920794-4657-4c34-80dd-dea0986491a0 nodeName:}" failed. No retries permitted until 2026-07-29 01:05:44.397154651 +0000 UTC m=+1774.706082473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e920794-4657-4c34-80dd-dea0986491a0-maas-api-tls") pod "maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d" (UID: "3e920794-4657-4c34-80dd-dea0986491a0") : secret "maas-api-e2e-derive-030a9f65-serving-cert" not found
Jul 29 01:05:43.898055 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.897908    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e920794-4657-4c34-80dd-dea0986491a0-openshift-service-ca\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:43.906482 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:43.906452    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnt2s\" (UniqueName: \"kubernetes.io/projected/3e920794-4657-4c34-80dd-dea0986491a0-kube-api-access-gnt2s\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:44.156328 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.156224    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:44.240751 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.240716    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:05:44.250062 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.250032    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"]
Jul 29 01:05:44.253939 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.253917    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:44.257444 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.257420    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-030a9f65\""
Jul 29 01:05:44.257557 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.257444    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-030a9f65-dockercfg-d6qbf\""
Jul 29 01:05:44.273741 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.273713    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"]
Jul 29 01:05:44.284098 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.284068    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"]
Jul 29 01:05:44.287544 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.287521    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:05:44.297622 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.297598    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"]
Jul 29 01:05:44.401866 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.401835    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358b107d-a28d-48c8-a8ba-e4e1ffa28802-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85\" (UID: \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:44.402057 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.401884    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdqk\" (UniqueName: \"kubernetes.io/projected/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-kube-api-access-5hdqk\") pod \"payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5\" (UID: \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\") " pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:05:44.402057 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.401965    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ndpr\" (UniqueName: \"kubernetes.io/projected/358b107d-a28d-48c8-a8ba-e4e1ffa28802-kube-api-access-8ndpr\") pod \"payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85\" (UID: \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:44.402057 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.402021    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-plugins-config-volume\") pod \"payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5\" (UID: \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\") " pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:05:44.402180 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.402071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e920794-4657-4c34-80dd-dea0986491a0-maas-api-tls\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:44.404640 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.404615    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e920794-4657-4c34-80dd-dea0986491a0-maas-api-tls\") pod \"maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:44.503378 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.503277    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ndpr\" (UniqueName: \"kubernetes.io/projected/358b107d-a28d-48c8-a8ba-e4e1ffa28802-kube-api-access-8ndpr\") pod \"payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85\" (UID: \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:44.503378 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.503338    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-plugins-config-volume\") pod \"payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5\" (UID: \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\") " pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:05:44.503597 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.503395    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358b107d-a28d-48c8-a8ba-e4e1ffa28802-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85\" (UID: \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:44.503597 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.503428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdqk\" (UniqueName: \"kubernetes.io/projected/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-kube-api-access-5hdqk\") pod \"payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5\" (UID: \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\") " pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:05:44.504368 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.504344    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358b107d-a28d-48c8-a8ba-e4e1ffa28802-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85\" (UID: \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:44.504425 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.504344    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-plugins-config-volume\") pod \"payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5\" (UID: \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\") " pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:05:44.513526 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.513490    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ndpr\" (UniqueName: \"kubernetes.io/projected/358b107d-a28d-48c8-a8ba-e4e1ffa28802-kube-api-access-8ndpr\") pod \"payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85\" (UID: \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:44.514322 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.514297    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdqk\" (UniqueName: \"kubernetes.io/projected/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-kube-api-access-5hdqk\") pod \"payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5\" (UID: \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\") " pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:05:44.532034 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.532000    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:44.565480 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.565443    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:44.600026 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.599813    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:05:44.713125 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.713100    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"]
Jul 29 01:05:44.744774 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.744741    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"]
Jul 29 01:05:44.746487 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:05:44.746460    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358b107d_a28d_48c8_a8ba_e4e1ffa28802.slice/crio-b0725dd6633b33ed5feb5a2084ac679dc7322ee99f57a69975646a25530e8e5e WatchSource:0}: Error finding container b0725dd6633b33ed5feb5a2084ac679dc7322ee99f57a69975646a25530e8e5e: Status 404 returned error can't find the container with id b0725dd6633b33ed5feb5a2084ac679dc7322ee99f57a69975646a25530e8e5e
Jul 29 01:05:44.768925 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:44.768900    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"]
Jul 29 01:05:44.769883 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:05:44.769851    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a23c4a1_52b8_4aed_b50d_ea1d2b157312.slice/crio-e1af1a0593658c880fef2b971cad21f507f127d2bda6bdf345fd7e260452a1c3 WatchSource:0}: Error finding container e1af1a0593658c880fef2b971cad21f507f127d2bda6bdf345fd7e260452a1c3: Status 404 returned error can't find the container with id e1af1a0593658c880fef2b971cad21f507f127d2bda6bdf345fd7e260452a1c3
Jul 29 01:05:45.622669 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.622628    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85" event={"ID":"358b107d-a28d-48c8-a8ba-e4e1ffa28802","Type":"ContainerStarted","Data":"4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8"}
Jul 29 01:05:45.623118 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.622695    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85" event={"ID":"358b107d-a28d-48c8-a8ba-e4e1ffa28802","Type":"ContainerStarted","Data":"b0725dd6633b33ed5feb5a2084ac679dc7322ee99f57a69975646a25530e8e5e"}
Jul 29 01:05:45.623118 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.622819    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:45.624347 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.624321    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d" event={"ID":"3e920794-4657-4c34-80dd-dea0986491a0","Type":"ContainerStarted","Data":"260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26"}
Jul 29 01:05:45.624503 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.624373    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:45.624503 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.624398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d" event={"ID":"3e920794-4657-4c34-80dd-dea0986491a0","Type":"ContainerStarted","Data":"000d02339a3999c367fd1c28a31d0d3b307a64ce8c73dbcb52f382dc06b5b338"}
Jul 29 01:05:45.625913 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.625888    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5" event={"ID":"2a23c4a1-52b8-4aed-b50d-ea1d2b157312","Type":"ContainerStarted","Data":"7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9"}
Jul 29 01:05:45.626033 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.625932    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5" event={"ID":"2a23c4a1-52b8-4aed-b50d-ea1d2b157312","Type":"ContainerStarted","Data":"e1af1a0593658c880fef2b971cad21f507f127d2bda6bdf345fd7e260452a1c3"}
Jul 29 01:05:45.626033 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.626000    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:05:45.643873 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.643817    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85" podStartSLOduration=1.643800213 podStartE2EDuration="1.643800213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:05:45.641381123 +0000 UTC m=+1775.950308966" watchObservedRunningTime="2026-07-29 01:05:45.643800213 +0000 UTC m=+1775.952728054"
Jul 29 01:05:45.660568 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.660505    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5" podStartSLOduration=1.6604842419999999 podStartE2EDuration="1.660484242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:05:45.659177163 +0000 UTC m=+1775.968105006" watchObservedRunningTime="2026-07-29 01:05:45.660484242 +0000 UTC m=+1775.969412085"
Jul 29 01:05:45.679570 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:45.679508    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d" podStartSLOduration=2.346861526 podStartE2EDuration="2.67949011s" totalImagesPullingTime="332.628584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:05:45.677235044 +0000 UTC m=+1775.986162886" watchObservedRunningTime="2026-07-29 01:05:45.67949011 +0000 UTC m=+1775.988417951"
Jul 29 01:05:51.638865 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:51.638831    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:05:56.635705 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:56.635674    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:05:56.636185 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:05:56.636005    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:06:08.370091 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:08.370030    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"]
Jul 29 01:06:08.374021 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:08.373994    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"
Jul 29 01:06:08.377232 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:08.377163    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:06:08.380966 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:08.380942    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"]
Jul 29 01:06:08.415469 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:08.415432    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz\" (UID: \"00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"
Jul 29 01:06:08.516396 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:08.516355    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz\" (UID: \"00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"
Jul 29 01:06:08.516994 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:08.516974    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz\" (UID: \"00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"
Jul 29 01:06:08.686679 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:08.686593    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"
Jul 29 01:06:08.830338 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:08.830311    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"]
Jul 29 01:06:08.832529 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:06:08.832496    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c1c03d_038d_4dc3_a5e7_bc0342fdcb5d.slice/crio-b6725ed1332b4b329e648f86d7b4fac9c9cabc2fac25d33de6dafbc742446b96 WatchSource:0}: Error finding container b6725ed1332b4b329e648f86d7b4fac9c9cabc2fac25d33de6dafbc742446b96: Status 404 returned error can't find the container with id b6725ed1332b4b329e648f86d7b4fac9c9cabc2fac25d33de6dafbc742446b96
Jul 29 01:06:09.768275 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:09.768231    2586 generic.go:356] "Generic (PLEG): container finished" podID="00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d" containerID="2350c0bd08e4c1c125d3cc0d00d050479492da65e5a4e714760f9ab9e10f2a5e" exitCode=0
Jul 29 01:06:09.768684 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:09.768294    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz" event={"ID":"00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d","Type":"ContainerDied","Data":"2350c0bd08e4c1c125d3cc0d00d050479492da65e5a4e714760f9ab9e10f2a5e"}
Jul 29 01:06:09.768684 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:09.768398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz" event={"ID":"00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d","Type":"ContainerStarted","Data":"b6725ed1332b4b329e648f86d7b4fac9c9cabc2fac25d33de6dafbc742446b96"}
Jul 29 01:06:10.913726 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:10.913696    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"
Jul 29 01:06:11.038606 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:11.038496    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d-maas-api-service-ca\") pod \"00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d\" (UID: \"00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d\") "
Jul 29 01:06:11.038878 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:11.038855    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d" (UID: "00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:06:11.139650 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:11.139606    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:06:11.780644 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:11.780616    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"
Jul 29 01:06:11.780836 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:11.780643    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz" event={"ID":"00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d","Type":"ContainerDied","Data":"b6725ed1332b4b329e648f86d7b4fac9c9cabc2fac25d33de6dafbc742446b96"}
Jul 29 01:06:11.780836 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:11.780690    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6725ed1332b4b329e648f86d7b4fac9c9cabc2fac25d33de6dafbc742446b96"
Jul 29 01:06:14.084244 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.084182    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"]
Jul 29 01:06:14.084720 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.084451    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d" podUID="3e920794-4657-4c34-80dd-dea0986491a0" containerName="maas-api" containerID="cri-o://260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26" gracePeriod=30
Jul 29 01:06:14.115540 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.112184    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:14.139738 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.139699    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"]
Jul 29 01:06:14.140282 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.139999    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5" podUID="2a23c4a1-52b8-4aed-b50d-ea1d2b157312" containerName="payload-processing" containerID="cri-o://7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9" gracePeriod=30
Jul 29 01:06:14.149052 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.149021    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"]
Jul 29 01:06:14.150092 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.149287    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85" podUID="358b107d-a28d-48c8-a8ba-e4e1ffa28802" containerName="payload-pre-processing" containerID="cri-o://4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8" gracePeriod=30
Jul 29 01:06:14.180480 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.180448    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:14.277005 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.276723    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"]
Jul 29 01:06:14.278584 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.278551    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:14.280375 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.280349    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030a9f65-lw8dz"]
Jul 29 01:06:14.421800 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.421766    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:06:14.463988 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.463878    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:06:14.471860 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.471836    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:06:14.471993 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.471941    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-plugins-config-volume\") pod \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\" (UID: \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\") "
Jul 29 01:06:14.472057 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.472045    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hdqk\" (UniqueName: \"kubernetes.io/projected/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-kube-api-access-5hdqk\") pod \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\" (UID: \"2a23c4a1-52b8-4aed-b50d-ea1d2b157312\") "
Jul 29 01:06:14.472294 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.472275    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a23c4a1-52b8-4aed-b50d-ea1d2b157312" (UID: "2a23c4a1-52b8-4aed-b50d-ea1d2b157312"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:06:14.473981 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.473959    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-kube-api-access-5hdqk" (OuterVolumeSpecName: "kube-api-access-5hdqk") pod "2a23c4a1-52b8-4aed-b50d-ea1d2b157312" (UID: "2a23c4a1-52b8-4aed-b50d-ea1d2b157312"). InnerVolumeSpecName "kube-api-access-5hdqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:06:14.572477 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.572444    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358b107d-a28d-48c8-a8ba-e4e1ffa28802-plugins-config-volume\") pod \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\" (UID: \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\") "
Jul 29 01:06:14.572649 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.572493    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnt2s\" (UniqueName: \"kubernetes.io/projected/3e920794-4657-4c34-80dd-dea0986491a0-kube-api-access-gnt2s\") pod \"3e920794-4657-4c34-80dd-dea0986491a0\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") "
Jul 29 01:06:14.572649 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.572532    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e920794-4657-4c34-80dd-dea0986491a0-openshift-service-ca\") pod \"3e920794-4657-4c34-80dd-dea0986491a0\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") "
Jul 29 01:06:14.572649 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.572598    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ndpr\" (UniqueName: \"kubernetes.io/projected/358b107d-a28d-48c8-a8ba-e4e1ffa28802-kube-api-access-8ndpr\") pod \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\" (UID: \"358b107d-a28d-48c8-a8ba-e4e1ffa28802\") "
Jul 29 01:06:14.572649 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.572639    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e920794-4657-4c34-80dd-dea0986491a0-maas-api-tls\") pod \"3e920794-4657-4c34-80dd-dea0986491a0\" (UID: \"3e920794-4657-4c34-80dd-dea0986491a0\") "
Jul 29 01:06:14.572868 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.572844    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:06:14.572921 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.572869    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hdqk\" (UniqueName: \"kubernetes.io/projected/2a23c4a1-52b8-4aed-b50d-ea1d2b157312-kube-api-access-5hdqk\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:06:14.572921 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.572891    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358b107d-a28d-48c8-a8ba-e4e1ffa28802-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "358b107d-a28d-48c8-a8ba-e4e1ffa28802" (UID: "358b107d-a28d-48c8-a8ba-e4e1ffa28802"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:06:14.572921 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.572908    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e920794-4657-4c34-80dd-dea0986491a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e920794-4657-4c34-80dd-dea0986491a0" (UID: "3e920794-4657-4c34-80dd-dea0986491a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:06:14.574768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.574737    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e920794-4657-4c34-80dd-dea0986491a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e920794-4657-4c34-80dd-dea0986491a0" (UID: "3e920794-4657-4c34-80dd-dea0986491a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:06:14.574851 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.574767    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e920794-4657-4c34-80dd-dea0986491a0-kube-api-access-gnt2s" (OuterVolumeSpecName: "kube-api-access-gnt2s") pod "3e920794-4657-4c34-80dd-dea0986491a0" (UID: "3e920794-4657-4c34-80dd-dea0986491a0"). InnerVolumeSpecName "kube-api-access-gnt2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:06:14.574851 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.574818    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358b107d-a28d-48c8-a8ba-e4e1ffa28802-kube-api-access-8ndpr" (OuterVolumeSpecName: "kube-api-access-8ndpr") pod "358b107d-a28d-48c8-a8ba-e4e1ffa28802" (UID: "358b107d-a28d-48c8-a8ba-e4e1ffa28802"). InnerVolumeSpecName "kube-api-access-8ndpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:06:14.674186 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.674077    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e920794-4657-4c34-80dd-dea0986491a0-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:06:14.674186 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.674116    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358b107d-a28d-48c8-a8ba-e4e1ffa28802-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:06:14.674186 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.674131    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnt2s\" (UniqueName: \"kubernetes.io/projected/3e920794-4657-4c34-80dd-dea0986491a0-kube-api-access-gnt2s\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:06:14.674186 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.674144    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e920794-4657-4c34-80dd-dea0986491a0-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:06:14.674186 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.674156    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ndpr\" (UniqueName: \"kubernetes.io/projected/358b107d-a28d-48c8-a8ba-e4e1ffa28802-kube-api-access-8ndpr\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:06:14.798013 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.797974    2586 generic.go:356] "Generic (PLEG): container finished" podID="358b107d-a28d-48c8-a8ba-e4e1ffa28802" containerID="4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8" exitCode=0
Jul 29 01:06:14.798234 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.798023    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"
Jul 29 01:06:14.798234 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.798062    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85" event={"ID":"358b107d-a28d-48c8-a8ba-e4e1ffa28802","Type":"ContainerDied","Data":"4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8"}
Jul 29 01:06:14.798234 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.798125    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85" event={"ID":"358b107d-a28d-48c8-a8ba-e4e1ffa28802","Type":"ContainerDied","Data":"b0725dd6633b33ed5feb5a2084ac679dc7322ee99f57a69975646a25530e8e5e"}
Jul 29 01:06:14.798234 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.798155    2586 scope.go:117] "RemoveContainer" containerID="4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8"
Jul 29 01:06:14.799424 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.799399    2586 generic.go:356] "Generic (PLEG): container finished" podID="3e920794-4657-4c34-80dd-dea0986491a0" containerID="260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26" exitCode=0
Jul 29 01:06:14.799533 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.799444    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"
Jul 29 01:06:14.799533 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.799498    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d" event={"ID":"3e920794-4657-4c34-80dd-dea0986491a0","Type":"ContainerDied","Data":"260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26"}
Jul 29 01:06:14.799624 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.799552    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d" event={"ID":"3e920794-4657-4c34-80dd-dea0986491a0","Type":"ContainerDied","Data":"000d02339a3999c367fd1c28a31d0d3b307a64ce8c73dbcb52f382dc06b5b338"}
Jul 29 01:06:14.801071 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.801045    2586 generic.go:356] "Generic (PLEG): container finished" podID="2a23c4a1-52b8-4aed-b50d-ea1d2b157312" containerID="7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9" exitCode=0
Jul 29 01:06:14.801183 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.801093    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"
Jul 29 01:06:14.801183 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.801095    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5" event={"ID":"2a23c4a1-52b8-4aed-b50d-ea1d2b157312","Type":"ContainerDied","Data":"7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9"}
Jul 29 01:06:14.801183 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.801171    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5" event={"ID":"2a23c4a1-52b8-4aed-b50d-ea1d2b157312","Type":"ContainerDied","Data":"e1af1a0593658c880fef2b971cad21f507f127d2bda6bdf345fd7e260452a1c3"}
Jul 29 01:06:14.809493 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.809469    2586 scope.go:117] "RemoveContainer" containerID="4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8"
Jul 29 01:06:14.809792 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:06:14.809772    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8\": container with ID starting with 4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8 not found: ID does not exist" containerID="4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8"
Jul 29 01:06:14.809860 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.809800    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8"} err="failed to get container status \"4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8\": rpc error: code = NotFound desc = could not find container \"4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8\": container with ID starting with 4f25e48a10b04184c6820a2dc4998a211e56f76a7c57753c3d4535e81e1b23b8 not found: ID does not exist"
Jul 29 01:06:14.809860 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.809816    2586 scope.go:117] "RemoveContainer" containerID="260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26"
Jul 29 01:06:14.819914 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.819897    2586 scope.go:117] "RemoveContainer" containerID="260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26"
Jul 29 01:06:14.820206 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:06:14.820175    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26\": container with ID starting with 260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26 not found: ID does not exist" containerID="260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26"
Jul 29 01:06:14.820255 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.820217    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26"} err="failed to get container status \"260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26\": rpc error: code = NotFound desc = could not find container \"260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26\": container with ID starting with 260c75df0f6b8121bb44f90cf4e38b1213e7a285344c1038e3fa3f87ba432e26 not found: ID does not exist"
Jul 29 01:06:14.820255 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.820238    2586 scope.go:117] "RemoveContainer" containerID="7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9"
Jul 29 01:06:14.828504 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.828483    2586 scope.go:117] "RemoveContainer" containerID="7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9"
Jul 29 01:06:14.828703 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.828681    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"]
Jul 29 01:06:14.828767 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:06:14.828738    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9\": container with ID starting with 7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9 not found: ID does not exist" containerID="7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9"
Jul 29 01:06:14.828811 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.828769    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9"} err="failed to get container status \"7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9\": rpc error: code = NotFound desc = could not find container \"7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9\": container with ID starting with 7802e351167ac986f3641fe609cb20611d5b73c39c73dcaeab639afdec1db2d9 not found: ID does not exist"
Jul 29 01:06:14.834483 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.834460    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030a9f65-7ddcf964d7-klv7d"]
Jul 29 01:06:14.847692 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.847660    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"]
Jul 29 01:06:14.852185 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.852137    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:14.854163 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.854140    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030a9f65-84c57675b4-82g85"]
Jul 29 01:06:14.864952 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.864919    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"]
Jul 29 01:06:14.870378 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.870349    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030a9f65-859fd4744b-d8hr5"]
Jul 29 01:06:14.917002 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:14.916973    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:15.139117 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:15.139089    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:15.230086 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:15.230054    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:15.332709 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:15.332676    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:16.221930 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:16.221891    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d" path="/var/lib/kubelet/pods/00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d/volumes"
Jul 29 01:06:16.222361 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:16.222281    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a23c4a1-52b8-4aed-b50d-ea1d2b157312" path="/var/lib/kubelet/pods/2a23c4a1-52b8-4aed-b50d-ea1d2b157312/volumes"
Jul 29 01:06:16.222594 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:16.222581    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358b107d-a28d-48c8-a8ba-e4e1ffa28802" path="/var/lib/kubelet/pods/358b107d-a28d-48c8-a8ba-e4e1ffa28802/volumes"
Jul 29 01:06:16.222921 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:16.222909    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e920794-4657-4c34-80dd-dea0986491a0" path="/var/lib/kubelet/pods/3e920794-4657-4c34-80dd-dea0986491a0/volumes"
Jul 29 01:06:16.660844 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:16.660813    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:23.184276 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:23.184233    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:23.291181 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:23.291137    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:23.387214 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:23.387170    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:23.485161 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:23.485069    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:23.878037 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:23.878002    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:33.973157 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:33.973116    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:34.073873 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:34.073836    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:49.941430 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:49.941381    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:56.160014 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:56.159983    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:56.344145 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:56.344113    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:06:56.444609 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:06:56.444522    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:12.215353 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:12.215303    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:18.050328 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:18.050289    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:18.114876 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:18.114844    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:18.190958 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:18.190924    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:18.289312 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:18.289254    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:28.788919 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:28.788872    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:29.514529 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:29.514493    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:35.660050 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:35.659962    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:35.764913 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:35.764873    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:35.859943 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:35.859909    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:35.959976 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:35.959884    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:41.883653 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:41.883622    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:41.937079 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:41.937049    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:42.002144 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:42.002108    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:43.763940 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:43.763907    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:49.385403 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:49.385355    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:49.484794 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:49.484762    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:50.045309 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:50.045276    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:56.170290 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:56.170253    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:56.274036 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:56.273970    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:56.349930 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:56.349894    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:56.528005 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:56.527887    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:59.725354 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.725268    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:59.811046 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811009    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"]
Jul 29 01:07:59.811576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811550    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358b107d-a28d-48c8-a8ba-e4e1ffa28802" containerName="payload-pre-processing"
Jul 29 01:07:59.811576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811573    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="358b107d-a28d-48c8-a8ba-e4e1ffa28802" containerName="payload-pre-processing"
Jul 29 01:07:59.811768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811586    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a23c4a1-52b8-4aed-b50d-ea1d2b157312" containerName="payload-processing"
Jul 29 01:07:59.811768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811595    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a23c4a1-52b8-4aed-b50d-ea1d2b157312" containerName="payload-processing"
Jul 29 01:07:59.811768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811630    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d" containerName="revoke-keys"
Jul 29 01:07:59.811768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811640    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d" containerName="revoke-keys"
Jul 29 01:07:59.811768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811652    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e920794-4657-4c34-80dd-dea0986491a0" containerName="maas-api"
Jul 29 01:07:59.811768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811688    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e920794-4657-4c34-80dd-dea0986491a0" containerName="maas-api"
Jul 29 01:07:59.811768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811766    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="358b107d-a28d-48c8-a8ba-e4e1ffa28802" containerName="payload-pre-processing"
Jul 29 01:07:59.812093 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811775    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a23c4a1-52b8-4aed-b50d-ea1d2b157312" containerName="payload-processing"
Jul 29 01:07:59.812093 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811785    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c1c03d-038d-4dc3-a5e7-bc0342fdcb5d" containerName="revoke-keys"
Jul 29 01:07:59.812093 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.811791    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e920794-4657-4c34-80dd-dea0986491a0" containerName="maas-api"
Jul 29 01:07:59.812383 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.812355    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:59.814889 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.814872    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:07:59.817568 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.817544    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b8aa042b-serving-cert\""
Jul 29 01:07:59.817806 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.817782    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 01:07:59.817916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.817805    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9jxwk\""
Jul 29 01:07:59.817916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.817805    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:07:59.826757 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.826728    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"]
Jul 29 01:07:59.900364 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.900326    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:07:59.912650 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.912613    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfa95ff-304b-412e-b6e8-27dc0376a987-openshift-service-ca\") pod \"maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:07:59.912814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.912682    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hgt\" (UniqueName: \"kubernetes.io/projected/ccfa95ff-304b-412e-b6e8-27dc0376a987-kube-api-access-k2hgt\") pod \"maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:07:59.912814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:07:59.912724    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfa95ff-304b-412e-b6e8-27dc0376a987-maas-api-tls\") pod \"maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:00.000549 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.000458    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:00.013608 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.013574    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hgt\" (UniqueName: \"kubernetes.io/projected/ccfa95ff-304b-412e-b6e8-27dc0376a987-kube-api-access-k2hgt\") pod \"maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:00.013803 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.013618    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfa95ff-304b-412e-b6e8-27dc0376a987-maas-api-tls\") pod \"maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:00.013803 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.013678    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfa95ff-304b-412e-b6e8-27dc0376a987-openshift-service-ca\") pod \"maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:00.014387 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.014367    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfa95ff-304b-412e-b6e8-27dc0376a987-openshift-service-ca\") pod \"maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:00.016031 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.016009    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfa95ff-304b-412e-b6e8-27dc0376a987-maas-api-tls\") pod \"maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:00.021361 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.021337    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hgt\" (UniqueName: \"kubernetes.io/projected/ccfa95ff-304b-412e-b6e8-27dc0376a987-kube-api-access-k2hgt\") pod \"maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:00.126547 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.126512    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:00.268577 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.268540    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"]
Jul 29 01:08:00.270847 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:08:00.270817    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfa95ff_304b_412e_b6e8_27dc0376a987.slice/crio-c46926877c76e7b282747e67cde9e9af39ad36cc687c110d0dc444e16f444f06 WatchSource:0}: Error finding container c46926877c76e7b282747e67cde9e9af39ad36cc687c110d0dc444e16f444f06: Status 404 returned error can't find the container with id c46926877c76e7b282747e67cde9e9af39ad36cc687c110d0dc444e16f444f06
Jul 29 01:08:00.411414 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.411377    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc" event={"ID":"ccfa95ff-304b-412e-b6e8-27dc0376a987","Type":"ContainerStarted","Data":"c46926877c76e7b282747e67cde9e9af39ad36cc687c110d0dc444e16f444f06"}
Jul 29 01:08:00.425656 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.425627    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:00.519828 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.519696    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:00.528307 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.528272    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"]
Jul 29 01:08:00.533821 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.533794    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:00.536380 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.536304    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b8aa042b\""
Jul 29 01:08:00.539504 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.539478    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"]
Jul 29 01:08:00.544868 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.544844    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"]
Jul 29 01:08:00.544984 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.544940    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:00.547497 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.547462    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b8aa042b-dockercfg-xnc7p\""
Jul 29 01:08:00.554491 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.554465    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"]
Jul 29 01:08:00.619558 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.619519    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4x6\" (UniqueName: \"kubernetes.io/projected/203c76fc-feb8-47a0-abce-ebd7457d9072-kube-api-access-mm4x6\") pod \"payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4\" (UID: \"203c76fc-feb8-47a0-abce-ebd7457d9072\") " pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:00.619738 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.619625    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0208a4a-b093-4d01-9705-914c2eeaf57f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv\" (UID: \"e0208a4a-b093-4d01-9705-914c2eeaf57f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:00.619738 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.619677    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203c76fc-feb8-47a0-abce-ebd7457d9072-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4\" (UID: \"203c76fc-feb8-47a0-abce-ebd7457d9072\") " pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:00.619738 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.619729    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrp4\" (UniqueName: \"kubernetes.io/projected/e0208a4a-b093-4d01-9705-914c2eeaf57f-kube-api-access-5vrp4\") pod \"payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv\" (UID: \"e0208a4a-b093-4d01-9705-914c2eeaf57f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:00.720876 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.720843    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4x6\" (UniqueName: \"kubernetes.io/projected/203c76fc-feb8-47a0-abce-ebd7457d9072-kube-api-access-mm4x6\") pod \"payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4\" (UID: \"203c76fc-feb8-47a0-abce-ebd7457d9072\") " pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:00.721054 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.720897    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0208a4a-b093-4d01-9705-914c2eeaf57f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv\" (UID: \"e0208a4a-b093-4d01-9705-914c2eeaf57f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:00.721054 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.720927    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203c76fc-feb8-47a0-abce-ebd7457d9072-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4\" (UID: \"203c76fc-feb8-47a0-abce-ebd7457d9072\") " pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:00.721054 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.720959    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrp4\" (UniqueName: \"kubernetes.io/projected/e0208a4a-b093-4d01-9705-914c2eeaf57f-kube-api-access-5vrp4\") pod \"payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv\" (UID: \"e0208a4a-b093-4d01-9705-914c2eeaf57f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:00.721705 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.721686    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0208a4a-b093-4d01-9705-914c2eeaf57f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv\" (UID: \"e0208a4a-b093-4d01-9705-914c2eeaf57f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:00.721752 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.721686    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203c76fc-feb8-47a0-abce-ebd7457d9072-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4\" (UID: \"203c76fc-feb8-47a0-abce-ebd7457d9072\") " pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:00.731373 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.731348    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrp4\" (UniqueName: \"kubernetes.io/projected/e0208a4a-b093-4d01-9705-914c2eeaf57f-kube-api-access-5vrp4\") pod \"payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv\" (UID: \"e0208a4a-b093-4d01-9705-914c2eeaf57f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:00.731696 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.731377    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4x6\" (UniqueName: \"kubernetes.io/projected/203c76fc-feb8-47a0-abce-ebd7457d9072-kube-api-access-mm4x6\") pod \"payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4\" (UID: \"203c76fc-feb8-47a0-abce-ebd7457d9072\") " pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:00.849183 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.849137    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:00.861711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:00.861604    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:01.011907 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.011586    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"]
Jul 29 01:08:01.014463 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:08:01.014421    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0208a4a_b093_4d01_9705_914c2eeaf57f.slice/crio-37180589fe7c68d846b4bff8f60915109c108bb10782c2a991fc9857dca7643b WatchSource:0}: Error finding container 37180589fe7c68d846b4bff8f60915109c108bb10782c2a991fc9857dca7643b: Status 404 returned error can't find the container with id 37180589fe7c68d846b4bff8f60915109c108bb10782c2a991fc9857dca7643b
Jul 29 01:08:01.042280 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.042251    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"]
Jul 29 01:08:01.043539 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:08:01.043512    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203c76fc_feb8_47a0_abce_ebd7457d9072.slice/crio-4e1b7342f9288309a8d739b181fd32e93f2bca7c5adc9c442c90c44891898671 WatchSource:0}: Error finding container 4e1b7342f9288309a8d739b181fd32e93f2bca7c5adc9c442c90c44891898671: Status 404 returned error can't find the container with id 4e1b7342f9288309a8d739b181fd32e93f2bca7c5adc9c442c90c44891898671
Jul 29 01:08:01.418469 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.418392    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv" event={"ID":"e0208a4a-b093-4d01-9705-914c2eeaf57f","Type":"ContainerStarted","Data":"a60f0baa4ea65a700a1b7898f9472c92a9e94fcafcf58d9a3343058a0ea81497"}
Jul 29 01:08:01.418645 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.418566    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:01.418645 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.418614    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv" event={"ID":"e0208a4a-b093-4d01-9705-914c2eeaf57f","Type":"ContainerStarted","Data":"37180589fe7c68d846b4bff8f60915109c108bb10782c2a991fc9857dca7643b"}
Jul 29 01:08:01.420120 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.420100    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4" event={"ID":"203c76fc-feb8-47a0-abce-ebd7457d9072","Type":"ContainerStarted","Data":"fe71aeb1ce59ce592b4f57e26d31e2e88565c3225ef8a4eede8a3d34e86bab1b"}
Jul 29 01:08:01.420235 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.420135    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4" event={"ID":"203c76fc-feb8-47a0-abce-ebd7457d9072","Type":"ContainerStarted","Data":"4e1b7342f9288309a8d739b181fd32e93f2bca7c5adc9c442c90c44891898671"}
Jul 29 01:08:01.420235 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.420176    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:01.436704 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.436653    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv" podStartSLOduration=1.43663681 podStartE2EDuration="1.43663681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:08:01.433126729 +0000 UTC m=+1911.742054572" watchObservedRunningTime="2026-07-29 01:08:01.43663681 +0000 UTC m=+1911.745564652"
Jul 29 01:08:01.454871 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:01.454807    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4" podStartSLOduration=1.454784644 podStartE2EDuration="1.454784644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:08:01.45194065 +0000 UTC m=+1911.760868501" watchObservedRunningTime="2026-07-29 01:08:01.454784644 +0000 UTC m=+1911.763712480"
Jul 29 01:08:02.427784 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:02.427743    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc" event={"ID":"ccfa95ff-304b-412e-b6e8-27dc0376a987","Type":"ContainerStarted","Data":"b0582744c1c2993109b3fb8525b374a16c5baa19bc123cb6560aeeb7bf5ee8a7"}
Jul 29 01:08:02.428179 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:02.428037    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:02.447899 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:02.447844    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc" podStartSLOduration=2.290981921 podStartE2EDuration="3.447828863s" totalImagesPullingTime="1.156846942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:08:02.446275003 +0000 UTC m=+1912.755202846" watchObservedRunningTime="2026-07-29 01:08:02.447828863 +0000 UTC m=+1912.756756704"
Jul 29 01:08:08.445867 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:08.445837    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:11.527519 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:11.527477    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:12.430702 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:12.430664    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:12.430880 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:12.430773    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:24.687016 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:24.686980    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:25.391178 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:25.391143    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:25.517121 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:25.517048    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:25.772103 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:25.772004    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:31.584114 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:31.584073    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:31.918937 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:31.918846    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"]
Jul 29 01:08:31.922847 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:31.922828    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"
Jul 29 01:08:31.925626 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:31.925599    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:08:31.929981 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:31.929946    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"]
Jul 29 01:08:31.985882 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:31.985841    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d207c3-8b5e-4022-ad77-42c62a68aef4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk\" (UID: \"43d207c3-8b5e-4022-ad77-42c62a68aef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"
Jul 29 01:08:32.086541 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:32.086507    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d207c3-8b5e-4022-ad77-42c62a68aef4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk\" (UID: \"43d207c3-8b5e-4022-ad77-42c62a68aef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"
Jul 29 01:08:32.087246 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:32.087223    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d207c3-8b5e-4022-ad77-42c62a68aef4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk\" (UID: \"43d207c3-8b5e-4022-ad77-42c62a68aef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"
Jul 29 01:08:32.234648 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:32.234555    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"
Jul 29 01:08:32.378691 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:32.378659    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"]
Jul 29 01:08:32.380995 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:08:32.380959    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d207c3_8b5e_4022_ad77_42c62a68aef4.slice/crio-dd4b1b5d2e9fac8c28fff26946c0ae44f54ea643e4da978a271968b6444687a3 WatchSource:0}: Error finding container dd4b1b5d2e9fac8c28fff26946c0ae44f54ea643e4da978a271968b6444687a3: Status 404 returned error can't find the container with id dd4b1b5d2e9fac8c28fff26946c0ae44f54ea643e4da978a271968b6444687a3
Jul 29 01:08:32.610803 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:32.610770    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk" event={"ID":"43d207c3-8b5e-4022-ad77-42c62a68aef4","Type":"ContainerStarted","Data":"dd4b1b5d2e9fac8c28fff26946c0ae44f54ea643e4da978a271968b6444687a3"}
Jul 29 01:08:33.616380 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:33.616341    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:33.620433 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:33.620406    2586 generic.go:356] "Generic (PLEG): container finished" podID="43d207c3-8b5e-4022-ad77-42c62a68aef4" containerID="2002795bf28a0bd90b77acc357c36ea5fcba967beaecd6b47bebaa104932bdc2" exitCode=0
Jul 29 01:08:33.620627 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:33.620456    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk" event={"ID":"43d207c3-8b5e-4022-ad77-42c62a68aef4","Type":"ContainerDied","Data":"2002795bf28a0bd90b77acc357c36ea5fcba967beaecd6b47bebaa104932bdc2"}
Jul 29 01:08:34.766788 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:34.766755    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"
Jul 29 01:08:34.809435 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:34.809398    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d207c3-8b5e-4022-ad77-42c62a68aef4-maas-api-service-ca\") pod \"43d207c3-8b5e-4022-ad77-42c62a68aef4\" (UID: \"43d207c3-8b5e-4022-ad77-42c62a68aef4\") "
Jul 29 01:08:34.809774 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:34.809749    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d207c3-8b5e-4022-ad77-42c62a68aef4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43d207c3-8b5e-4022-ad77-42c62a68aef4" (UID: "43d207c3-8b5e-4022-ad77-42c62a68aef4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:08:34.910531 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:34.910433    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d207c3-8b5e-4022-ad77-42c62a68aef4-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:08:35.632923 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:35.632889    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"
Jul 29 01:08:35.632923 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:35.632916    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk" event={"ID":"43d207c3-8b5e-4022-ad77-42c62a68aef4","Type":"ContainerDied","Data":"dd4b1b5d2e9fac8c28fff26946c0ae44f54ea643e4da978a271968b6444687a3"}
Jul 29 01:08:35.633160 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:35.632968    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd4b1b5d2e9fac8c28fff26946c0ae44f54ea643e4da978a271968b6444687a3"
Jul 29 01:08:42.363438 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.363393    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"]
Jul 29 01:08:42.364010 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.363662    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc" podUID="ccfa95ff-304b-412e-b6e8-27dc0376a987" containerName="maas-api" containerID="cri-o://b0582744c1c2993109b3fb8525b374a16c5baa19bc123cb6560aeeb7bf5ee8a7" gracePeriod=30
Jul 29 01:08:42.392923 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.392886    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:42.415335 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.415298    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"]
Jul 29 01:08:42.415593 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.415568    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4" podUID="203c76fc-feb8-47a0-abce-ebd7457d9072" containerName="payload-processing" containerID="cri-o://fe71aeb1ce59ce592b4f57e26d31e2e88565c3225ef8a4eede8a3d34e86bab1b" gracePeriod=30
Jul 29 01:08:42.420865 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.419813    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"]
Jul 29 01:08:42.420865 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.420148    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv" podUID="e0208a4a-b093-4d01-9705-914c2eeaf57f" containerName="payload-pre-processing" containerID="cri-o://a60f0baa4ea65a700a1b7898f9472c92a9e94fcafcf58d9a3343058a0ea81497" gracePeriod=30
Jul 29 01:08:42.428561 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.428525    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4" podUID="203c76fc-feb8-47a0-abce-ebd7457d9072" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.71:9004: connect: connection refused"
Jul 29 01:08:42.428704 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.428668    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv" podUID="e0208a4a-b093-4d01-9705-914c2eeaf57f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.70:9004: connect: connection refused"
Jul 29 01:08:42.493923 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.493890    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:42.542129 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.542071    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"]
Jul 29 01:08:42.545287 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.545260    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8aa042b-stbbk"]
Jul 29 01:08:42.679925 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.679876    2586 generic.go:356] "Generic (PLEG): container finished" podID="e0208a4a-b093-4d01-9705-914c2eeaf57f" containerID="a60f0baa4ea65a700a1b7898f9472c92a9e94fcafcf58d9a3343058a0ea81497" exitCode=0
Jul 29 01:08:42.680115 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.679999    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv" event={"ID":"e0208a4a-b093-4d01-9705-914c2eeaf57f","Type":"ContainerDied","Data":"a60f0baa4ea65a700a1b7898f9472c92a9e94fcafcf58d9a3343058a0ea81497"}
Jul 29 01:08:42.682987 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.682956    2586 generic.go:356] "Generic (PLEG): container finished" podID="203c76fc-feb8-47a0-abce-ebd7457d9072" containerID="fe71aeb1ce59ce592b4f57e26d31e2e88565c3225ef8a4eede8a3d34e86bab1b" exitCode=0
Jul 29 01:08:42.683116 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.683081    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4" event={"ID":"203c76fc-feb8-47a0-abce-ebd7457d9072","Type":"ContainerDied","Data":"fe71aeb1ce59ce592b4f57e26d31e2e88565c3225ef8a4eede8a3d34e86bab1b"}
Jul 29 01:08:42.684743 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.684700    2586 generic.go:356] "Generic (PLEG): container finished" podID="ccfa95ff-304b-412e-b6e8-27dc0376a987" containerID="b0582744c1c2993109b3fb8525b374a16c5baa19bc123cb6560aeeb7bf5ee8a7" exitCode=0
Jul 29 01:08:42.684899 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.684786    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc" event={"ID":"ccfa95ff-304b-412e-b6e8-27dc0376a987","Type":"ContainerDied","Data":"b0582744c1c2993109b3fb8525b374a16c5baa19bc123cb6560aeeb7bf5ee8a7"}
Jul 29 01:08:42.684899 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.684831    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc" event={"ID":"ccfa95ff-304b-412e-b6e8-27dc0376a987","Type":"ContainerDied","Data":"c46926877c76e7b282747e67cde9e9af39ad36cc687c110d0dc444e16f444f06"}
Jul 29 01:08:42.684899 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.684855    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46926877c76e7b282747e67cde9e9af39ad36cc687c110d0dc444e16f444f06"
Jul 29 01:08:42.688935 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.688908    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:42.720378 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.720347    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:42.730593 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.730566    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:42.780340 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.780292    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfa95ff-304b-412e-b6e8-27dc0376a987-openshift-service-ca\") pod \"ccfa95ff-304b-412e-b6e8-27dc0376a987\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") "
Jul 29 01:08:42.780340 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.780343    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm4x6\" (UniqueName: \"kubernetes.io/projected/203c76fc-feb8-47a0-abce-ebd7457d9072-kube-api-access-mm4x6\") pod \"203c76fc-feb8-47a0-abce-ebd7457d9072\" (UID: \"203c76fc-feb8-47a0-abce-ebd7457d9072\") "
Jul 29 01:08:42.780586 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.780368    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2hgt\" (UniqueName: \"kubernetes.io/projected/ccfa95ff-304b-412e-b6e8-27dc0376a987-kube-api-access-k2hgt\") pod \"ccfa95ff-304b-412e-b6e8-27dc0376a987\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") "
Jul 29 01:08:42.780586 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.780385    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfa95ff-304b-412e-b6e8-27dc0376a987-maas-api-tls\") pod \"ccfa95ff-304b-412e-b6e8-27dc0376a987\" (UID: \"ccfa95ff-304b-412e-b6e8-27dc0376a987\") "
Jul 29 01:08:42.780586 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.780402    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203c76fc-feb8-47a0-abce-ebd7457d9072-plugins-config-volume\") pod \"203c76fc-feb8-47a0-abce-ebd7457d9072\" (UID: \"203c76fc-feb8-47a0-abce-ebd7457d9072\") "
Jul 29 01:08:42.780586 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.780421    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vrp4\" (UniqueName: \"kubernetes.io/projected/e0208a4a-b093-4d01-9705-914c2eeaf57f-kube-api-access-5vrp4\") pod \"e0208a4a-b093-4d01-9705-914c2eeaf57f\" (UID: \"e0208a4a-b093-4d01-9705-914c2eeaf57f\") "
Jul 29 01:08:42.780586 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.780467    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0208a4a-b093-4d01-9705-914c2eeaf57f-plugins-config-volume\") pod \"e0208a4a-b093-4d01-9705-914c2eeaf57f\" (UID: \"e0208a4a-b093-4d01-9705-914c2eeaf57f\") "
Jul 29 01:08:42.780855 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.780808    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203c76fc-feb8-47a0-abce-ebd7457d9072-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "203c76fc-feb8-47a0-abce-ebd7457d9072" (UID: "203c76fc-feb8-47a0-abce-ebd7457d9072"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:08:42.780976 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.780940    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfa95ff-304b-412e-b6e8-27dc0376a987-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ccfa95ff-304b-412e-b6e8-27dc0376a987" (UID: "ccfa95ff-304b-412e-b6e8-27dc0376a987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:08:42.781315 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.781266    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0208a4a-b093-4d01-9705-914c2eeaf57f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0208a4a-b093-4d01-9705-914c2eeaf57f" (UID: "e0208a4a-b093-4d01-9705-914c2eeaf57f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:08:42.783261 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.783228    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0208a4a-b093-4d01-9705-914c2eeaf57f-kube-api-access-5vrp4" (OuterVolumeSpecName: "kube-api-access-5vrp4") pod "e0208a4a-b093-4d01-9705-914c2eeaf57f" (UID: "e0208a4a-b093-4d01-9705-914c2eeaf57f"). InnerVolumeSpecName "kube-api-access-5vrp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:08:42.783350 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.783259    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfa95ff-304b-412e-b6e8-27dc0376a987-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ccfa95ff-304b-412e-b6e8-27dc0376a987" (UID: "ccfa95ff-304b-412e-b6e8-27dc0376a987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:08:42.783350 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.783273    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfa95ff-304b-412e-b6e8-27dc0376a987-kube-api-access-k2hgt" (OuterVolumeSpecName: "kube-api-access-k2hgt") pod "ccfa95ff-304b-412e-b6e8-27dc0376a987" (UID: "ccfa95ff-304b-412e-b6e8-27dc0376a987"). InnerVolumeSpecName "kube-api-access-k2hgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:08:42.783350 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.783277    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203c76fc-feb8-47a0-abce-ebd7457d9072-kube-api-access-mm4x6" (OuterVolumeSpecName: "kube-api-access-mm4x6") pod "203c76fc-feb8-47a0-abce-ebd7457d9072" (UID: "203c76fc-feb8-47a0-abce-ebd7457d9072"). InnerVolumeSpecName "kube-api-access-mm4x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:08:42.881881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.881793    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfa95ff-304b-412e-b6e8-27dc0376a987-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:08:42.881881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.881822    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4x6\" (UniqueName: \"kubernetes.io/projected/203c76fc-feb8-47a0-abce-ebd7457d9072-kube-api-access-mm4x6\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:08:42.881881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.881833    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hgt\" (UniqueName: \"kubernetes.io/projected/ccfa95ff-304b-412e-b6e8-27dc0376a987-kube-api-access-k2hgt\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:08:42.881881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.881843    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfa95ff-304b-412e-b6e8-27dc0376a987-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:08:42.881881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.881853    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203c76fc-feb8-47a0-abce-ebd7457d9072-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:08:42.881881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.881864    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vrp4\" (UniqueName: \"kubernetes.io/projected/e0208a4a-b093-4d01-9705-914c2eeaf57f-kube-api-access-5vrp4\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:08:42.881881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:42.881873    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0208a4a-b093-4d01-9705-914c2eeaf57f-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:08:43.124033 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.123995    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:43.224810 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.224727    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:43.697660 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.697621    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"
Jul 29 01:08:43.697660 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.697647    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4" event={"ID":"203c76fc-feb8-47a0-abce-ebd7457d9072","Type":"ContainerDied","Data":"4e1b7342f9288309a8d739b181fd32e93f2bca7c5adc9c442c90c44891898671"}
Jul 29 01:08:43.698235 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.697749    2586 scope.go:117] "RemoveContainer" containerID="fe71aeb1ce59ce592b4f57e26d31e2e88565c3225ef8a4eede8a3d34e86bab1b"
Jul 29 01:08:43.699319 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.699292    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"
Jul 29 01:08:43.699319 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.699309    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv" event={"ID":"e0208a4a-b093-4d01-9705-914c2eeaf57f","Type":"ContainerDied","Data":"37180589fe7c68d846b4bff8f60915109c108bb10782c2a991fc9857dca7643b"}
Jul 29 01:08:43.699508 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.699302    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"
Jul 29 01:08:43.709407 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.709385    2586 scope.go:117] "RemoveContainer" containerID="a60f0baa4ea65a700a1b7898f9472c92a9e94fcafcf58d9a3343058a0ea81497"
Jul 29 01:08:43.725570 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.725536    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"]
Jul 29 01:08:43.731208 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.731163    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8aa042b-54d989b6cb-4zhfv"]
Jul 29 01:08:43.741326 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.741293    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"]
Jul 29 01:08:43.744766 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.744736    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8aa042b-8bd6cd6bf-qqfsc"]
Jul 29 01:08:43.756644 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.756609    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"]
Jul 29 01:08:43.758837 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:43.758810    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8aa042b-54c6db6b86-tqcz4"]
Jul 29 01:08:44.221892 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:44.221855    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="203c76fc-feb8-47a0-abce-ebd7457d9072" path="/var/lib/kubelet/pods/203c76fc-feb8-47a0-abce-ebd7457d9072/volumes"
Jul 29 01:08:44.222218 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:44.222185    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d207c3-8b5e-4022-ad77-42c62a68aef4" path="/var/lib/kubelet/pods/43d207c3-8b5e-4022-ad77-42c62a68aef4/volumes"
Jul 29 01:08:44.222569 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:44.222556    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccfa95ff-304b-412e-b6e8-27dc0376a987" path="/var/lib/kubelet/pods/ccfa95ff-304b-412e-b6e8-27dc0376a987/volumes"
Jul 29 01:08:44.222865 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:44.222854    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0208a4a-b093-4d01-9705-914c2eeaf57f" path="/var/lib/kubelet/pods/e0208a4a-b093-4d01-9705-914c2eeaf57f/volumes"
Jul 29 01:08:48.612493 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:48.612462    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:48.713405 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:48.713369    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:48.873166 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:48.873077    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:49.105951 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:49.105917    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:49.282814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:49.282740    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:49.374950 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:49.374917    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:49.474716 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:49.474683    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:55.111647 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:55.111610    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:55.174665 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:55.174628    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:55.263623 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:55.263591    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:55.363612 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:55.363516    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:55.469565 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:55.469519    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:08:56.982916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:08:56.982884    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:00.962252 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:00.962184    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:01.063348 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:01.063314    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:07.754588 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:07.754549    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:08.057312 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:08.057219    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:08.269336 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:08.269297    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:11.074369 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.074337    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:11.148854 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.148820    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:11.155309 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155283    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"]
Jul 29 01:09:11.155729 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155708    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfa95ff-304b-412e-b6e8-27dc0376a987" containerName="maas-api"
Jul 29 01:09:11.155814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155732    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfa95ff-304b-412e-b6e8-27dc0376a987" containerName="maas-api"
Jul 29 01:09:11.155814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155747    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0208a4a-b093-4d01-9705-914c2eeaf57f" containerName="payload-pre-processing"
Jul 29 01:09:11.155814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155755    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0208a4a-b093-4d01-9705-914c2eeaf57f" containerName="payload-pre-processing"
Jul 29 01:09:11.155814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155773    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203c76fc-feb8-47a0-abce-ebd7457d9072" containerName="payload-processing"
Jul 29 01:09:11.155814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155782    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="203c76fc-feb8-47a0-abce-ebd7457d9072" containerName="payload-processing"
Jul 29 01:09:11.155814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155798    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d207c3-8b5e-4022-ad77-42c62a68aef4" containerName="revoke-keys"
Jul 29 01:09:11.155814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155807    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d207c3-8b5e-4022-ad77-42c62a68aef4" containerName="revoke-keys"
Jul 29 01:09:11.156143 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155900    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfa95ff-304b-412e-b6e8-27dc0376a987" containerName="maas-api"
Jul 29 01:09:11.156143 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155914    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0208a4a-b093-4d01-9705-914c2eeaf57f" containerName="payload-pre-processing"
Jul 29 01:09:11.156143 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155927    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="203c76fc-feb8-47a0-abce-ebd7457d9072" containerName="payload-processing"
Jul 29 01:09:11.156143 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.155936    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d207c3-8b5e-4022-ad77-42c62a68aef4" containerName="revoke-keys"
Jul 29 01:09:11.160665 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.160645    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.163516 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.163497    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:09:11.163617 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.163496    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9bb5bd21-serving-cert\""
Jul 29 01:09:11.164660 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.164640    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9jxwk\""
Jul 29 01:09:11.164660 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.164655    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 01:09:11.169216 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.169175    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"]
Jul 29 01:09:11.224377 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.224347    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:11.318262 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.318226    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:11.350574 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.350480    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4hqw\" (UniqueName: \"kubernetes.io/projected/2fec67b0-77a9-4092-846c-7c124c247f27-kube-api-access-p4hqw\") pod \"maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.350574 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.350538    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fec67b0-77a9-4092-846c-7c124c247f27-maas-api-tls\") pod \"maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.350826 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.350616    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fec67b0-77a9-4092-846c-7c124c247f27-openshift-service-ca\") pod \"maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.452021 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.451981    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4hqw\" (UniqueName: \"kubernetes.io/projected/2fec67b0-77a9-4092-846c-7c124c247f27-kube-api-access-p4hqw\") pod \"maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.452021 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.452028    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fec67b0-77a9-4092-846c-7c124c247f27-maas-api-tls\") pod \"maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.452338 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.452070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fec67b0-77a9-4092-846c-7c124c247f27-openshift-service-ca\") pod \"maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.452897 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.452875    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fec67b0-77a9-4092-846c-7c124c247f27-openshift-service-ca\") pod \"maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.454682 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.454660    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fec67b0-77a9-4092-846c-7c124c247f27-maas-api-tls\") pod \"maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.460727 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.460699    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4hqw\" (UniqueName: \"kubernetes.io/projected/2fec67b0-77a9-4092-846c-7c124c247f27-kube-api-access-p4hqw\") pod \"maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.472610 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.472581    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:11.639876 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:09:11.639843    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fec67b0_77a9_4092_846c_7c124c247f27.slice/crio-6e3e23c003d9b17975b1c3659383c3ad083de30e4833b7a3511054b23b66bca6 WatchSource:0}: Error finding container 6e3e23c003d9b17975b1c3659383c3ad083de30e4833b7a3511054b23b66bca6: Status 404 returned error can't find the container with id 6e3e23c003d9b17975b1c3659383c3ad083de30e4833b7a3511054b23b66bca6
Jul 29 01:09:11.640482 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.640453    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"]
Jul 29 01:09:11.843170 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.843129    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:11.865692 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.865650    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq" event={"ID":"2fec67b0-77a9-4092-846c-7c124c247f27","Type":"ContainerStarted","Data":"6e3e23c003d9b17975b1c3659383c3ad083de30e4833b7a3511054b23b66bca6"}
Jul 29 01:09:11.939215 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.939113    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:11.959031 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.958993    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"]
Jul 29 01:09:11.963818 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.963794    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:11.967406 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.967378    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9bb5bd21\""
Jul 29 01:09:11.967517 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.967454    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9bb5bd21-dockercfg-785wx\""
Jul 29 01:09:11.973223 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.973180    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"]
Jul 29 01:09:11.985632 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.985588    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"]
Jul 29 01:09:11.989655 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.989628    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:11.999527 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:11.999496    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"]
Jul 29 01:09:12.058830 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.058788    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdcda35-70ed-489d-8831-d31b75421bc4-plugins-config-volume\") pod \"payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn\" (UID: \"1cdcda35-70ed-489d-8831-d31b75421bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:12.059004 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.058929    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f65j\" (UniqueName: \"kubernetes.io/projected/1cdcda35-70ed-489d-8831-d31b75421bc4-kube-api-access-6f65j\") pod \"payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn\" (UID: \"1cdcda35-70ed-489d-8831-d31b75421bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:12.160015 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.159970    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc46774-9863-45f5-9ad2-6053aed86acd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv\" (UID: \"7bc46774-9863-45f5-9ad2-6053aed86acd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:12.160489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.160058    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f65j\" (UniqueName: \"kubernetes.io/projected/1cdcda35-70ed-489d-8831-d31b75421bc4-kube-api-access-6f65j\") pod \"payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn\" (UID: \"1cdcda35-70ed-489d-8831-d31b75421bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:12.160489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.160117    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjz7\" (UniqueName: \"kubernetes.io/projected/7bc46774-9863-45f5-9ad2-6053aed86acd-kube-api-access-hcjz7\") pod \"payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv\" (UID: \"7bc46774-9863-45f5-9ad2-6053aed86acd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:12.160489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.160183    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdcda35-70ed-489d-8831-d31b75421bc4-plugins-config-volume\") pod \"payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn\" (UID: \"1cdcda35-70ed-489d-8831-d31b75421bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:12.161124 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.161097    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdcda35-70ed-489d-8831-d31b75421bc4-plugins-config-volume\") pod \"payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn\" (UID: \"1cdcda35-70ed-489d-8831-d31b75421bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:12.168503 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.168467    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f65j\" (UniqueName: \"kubernetes.io/projected/1cdcda35-70ed-489d-8831-d31b75421bc4-kube-api-access-6f65j\") pod \"payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn\" (UID: \"1cdcda35-70ed-489d-8831-d31b75421bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:12.261066 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.260672    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc46774-9863-45f5-9ad2-6053aed86acd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv\" (UID: \"7bc46774-9863-45f5-9ad2-6053aed86acd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:12.261066 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.260794    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjz7\" (UniqueName: \"kubernetes.io/projected/7bc46774-9863-45f5-9ad2-6053aed86acd-kube-api-access-hcjz7\") pod \"payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv\" (UID: \"7bc46774-9863-45f5-9ad2-6053aed86acd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:12.261809 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.261787    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc46774-9863-45f5-9ad2-6053aed86acd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv\" (UID: \"7bc46774-9863-45f5-9ad2-6053aed86acd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:12.268587 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.268564    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjz7\" (UniqueName: \"kubernetes.io/projected/7bc46774-9863-45f5-9ad2-6053aed86acd-kube-api-access-hcjz7\") pod \"payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv\" (UID: \"7bc46774-9863-45f5-9ad2-6053aed86acd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:12.277778 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.277754    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:12.303376 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.303347    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:12.438020 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.437954    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"]
Jul 29 01:09:12.439375 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:09:12.439341    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc46774_9863_45f5_9ad2_6053aed86acd.slice/crio-620a892dd69255c277eda95156beb8303b062a49870d0c79556e7bf5db0a1210 WatchSource:0}: Error finding container 620a892dd69255c277eda95156beb8303b062a49870d0c79556e7bf5db0a1210: Status 404 returned error can't find the container with id 620a892dd69255c277eda95156beb8303b062a49870d0c79556e7bf5db0a1210
Jul 29 01:09:12.467713 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.467641    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"]
Jul 29 01:09:12.468330 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:09:12.468296    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdcda35_70ed_489d_8831_d31b75421bc4.slice/crio-f6e2c5d1455741d2c499d92d702721e3bc55ae84339bce378fe7b3b217253d2b WatchSource:0}: Error finding container f6e2c5d1455741d2c499d92d702721e3bc55ae84339bce378fe7b3b217253d2b: Status 404 returned error can't find the container with id f6e2c5d1455741d2c499d92d702721e3bc55ae84339bce378fe7b3b217253d2b
Jul 29 01:09:12.873389 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.873345    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq" event={"ID":"2fec67b0-77a9-4092-846c-7c124c247f27","Type":"ContainerStarted","Data":"1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d"}
Jul 29 01:09:12.873598 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.873480    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:12.875212 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.875162    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn" event={"ID":"1cdcda35-70ed-489d-8831-d31b75421bc4","Type":"ContainerStarted","Data":"14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4"}
Jul 29 01:09:12.875350 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.875230    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn" event={"ID":"1cdcda35-70ed-489d-8831-d31b75421bc4","Type":"ContainerStarted","Data":"f6e2c5d1455741d2c499d92d702721e3bc55ae84339bce378fe7b3b217253d2b"}
Jul 29 01:09:12.875350 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.875261    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:12.876889 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.876861    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv" event={"ID":"7bc46774-9863-45f5-9ad2-6053aed86acd","Type":"ContainerStarted","Data":"0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae"}
Jul 29 01:09:12.877046 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.876909    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv" event={"ID":"7bc46774-9863-45f5-9ad2-6053aed86acd","Type":"ContainerStarted","Data":"620a892dd69255c277eda95156beb8303b062a49870d0c79556e7bf5db0a1210"}
Jul 29 01:09:12.877046 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.876957    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:12.889142 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.888912    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq" podStartSLOduration=1.31333232 podStartE2EDuration="1.888894085s" totalImagesPullingTime="575.561765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:09:12.888557384 +0000 UTC m=+1983.197485239" watchObservedRunningTime="2026-07-29 01:09:12.888894085 +0000 UTC m=+1983.197821928"
Jul 29 01:09:12.905835 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.905775    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn" podStartSLOduration=1.905756893 podStartE2EDuration="1.905756893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:09:12.903720175 +0000 UTC m=+1983.212648020" watchObservedRunningTime="2026-07-29 01:09:12.905756893 +0000 UTC m=+1983.214684735"
Jul 29 01:09:12.922775 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:12.922714    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv" podStartSLOduration=1.922697141 podStartE2EDuration="1.922697141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:09:12.92018386 +0000 UTC m=+1983.229111705" watchObservedRunningTime="2026-07-29 01:09:12.922697141 +0000 UTC m=+1983.231624983"
Jul 29 01:09:18.888762 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:18.888732    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:09:23.884646 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:23.884616    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:09:23.885076 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:23.884689    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:09:36.030443 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:36.030412    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:09:40.444516 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:09:40.444475    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:10:05.169625 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:05.169586    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"]
Jul 29 01:10:05.173498 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:05.173471    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"
Jul 29 01:10:05.176244 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:05.176219    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:10:05.181488 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:05.181459    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"]
Jul 29 01:10:05.292162 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:05.292125    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558ef52f-d1c1-49e1-819d-16ac3f82a670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5\" (UID: \"558ef52f-d1c1-49e1-819d-16ac3f82a670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"
Jul 29 01:10:05.393084 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:05.393041    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558ef52f-d1c1-49e1-819d-16ac3f82a670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5\" (UID: \"558ef52f-d1c1-49e1-819d-16ac3f82a670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"
Jul 29 01:10:05.393721 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:05.393700    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558ef52f-d1c1-49e1-819d-16ac3f82a670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5\" (UID: \"558ef52f-d1c1-49e1-819d-16ac3f82a670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"
Jul 29 01:10:05.487671 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:05.487575    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"
Jul 29 01:10:05.631012 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:05.630980    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"]
Jul 29 01:10:05.632129 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:10:05.632098    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558ef52f_d1c1_49e1_819d_16ac3f82a670.slice/crio-6bccc87e60045b6557f307844c04b06caf22e554eaef9928efac6475c704de6f WatchSource:0}: Error finding container 6bccc87e60045b6557f307844c04b06caf22e554eaef9928efac6475c704de6f: Status 404 returned error can't find the container with id 6bccc87e60045b6557f307844c04b06caf22e554eaef9928efac6475c704de6f
Jul 29 01:10:06.198521 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:06.198426    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5" event={"ID":"558ef52f-d1c1-49e1-819d-16ac3f82a670","Type":"ContainerStarted","Data":"34ed6ae6cc7c2b494fd46366551c3b8e2152d578b4b812837222d4e2b5642668"}
Jul 29 01:10:06.198521 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:06.198480    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5" event={"ID":"558ef52f-d1c1-49e1-819d-16ac3f82a670","Type":"ContainerStarted","Data":"6bccc87e60045b6557f307844c04b06caf22e554eaef9928efac6475c704de6f"}
Jul 29 01:10:06.214495 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:06.214431    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5" podStartSLOduration=1.21441478 podStartE2EDuration="1.21441478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:06.212041665 +0000 UTC m=+2036.520969507" watchObservedRunningTime="2026-07-29 01:10:06.21441478 +0000 UTC m=+2036.523342636"
Jul 29 01:10:07.206442 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:07.206340    2586 generic.go:356] "Generic (PLEG): container finished" podID="558ef52f-d1c1-49e1-819d-16ac3f82a670" containerID="34ed6ae6cc7c2b494fd46366551c3b8e2152d578b4b812837222d4e2b5642668" exitCode=0
Jul 29 01:10:07.206819 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:07.206431    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5" event={"ID":"558ef52f-d1c1-49e1-819d-16ac3f82a670","Type":"ContainerDied","Data":"34ed6ae6cc7c2b494fd46366551c3b8e2152d578b4b812837222d4e2b5642668"}
Jul 29 01:10:08.351475 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:08.351447    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"
Jul 29 01:10:08.422060 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:08.422018    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558ef52f-d1c1-49e1-819d-16ac3f82a670-maas-api-service-ca\") pod \"558ef52f-d1c1-49e1-819d-16ac3f82a670\" (UID: \"558ef52f-d1c1-49e1-819d-16ac3f82a670\") "
Jul 29 01:10:08.422404 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:08.422379    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558ef52f-d1c1-49e1-819d-16ac3f82a670-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "558ef52f-d1c1-49e1-819d-16ac3f82a670" (UID: "558ef52f-d1c1-49e1-819d-16ac3f82a670"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:10:08.523231 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:08.523119    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558ef52f-d1c1-49e1-819d-16ac3f82a670-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:10:09.225092 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:09.225059    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"
Jul 29 01:10:09.225283 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:09.225085    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5" event={"ID":"558ef52f-d1c1-49e1-819d-16ac3f82a670","Type":"ContainerDied","Data":"6bccc87e60045b6557f307844c04b06caf22e554eaef9928efac6475c704de6f"}
Jul 29 01:10:09.225283 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:09.225157    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bccc87e60045b6557f307844c04b06caf22e554eaef9928efac6475c704de6f"
Jul 29 01:10:20.578789 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.578752    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"]
Jul 29 01:10:20.579357 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.579008    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq" podUID="2fec67b0-77a9-4092-846c-7c124c247f27" containerName="maas-api" containerID="cri-o://1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d" gracePeriod=30
Jul 29 01:10:20.602099 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.602028    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:10:20.631133 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.631096    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"]
Jul 29 01:10:20.631455 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.631399    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn" podUID="1cdcda35-70ed-489d-8831-d31b75421bc4" containerName="payload-processing" containerID="cri-o://14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4" gracePeriod=30
Jul 29 01:10:20.640272 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.639832    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"]
Jul 29 01:10:20.640272 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.640090    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv" podUID="7bc46774-9863-45f5-9ad2-6053aed86acd" containerName="payload-pre-processing" containerID="cri-o://0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae" gracePeriod=30
Jul 29 01:10:20.684189 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.684155    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:10:20.761143 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.761112    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:10:20.761831 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.761813    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"]
Jul 29 01:10:20.766362 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.766333    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bb5bd21-5gxw5"]
Jul 29 01:10:20.822864 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.822818    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:10:20.924882 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.924858    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:10:20.951220 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.951169    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:10:20.954341 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:20.954317    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:10:21.032051 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.032010    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fec67b0-77a9-4092-846c-7c124c247f27-openshift-service-ca\") pod \"2fec67b0-77a9-4092-846c-7c124c247f27\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") "
Jul 29 01:10:21.032213 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.032078    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4hqw\" (UniqueName: \"kubernetes.io/projected/2fec67b0-77a9-4092-846c-7c124c247f27-kube-api-access-p4hqw\") pod \"2fec67b0-77a9-4092-846c-7c124c247f27\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") "
Jul 29 01:10:21.032213 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.032113    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fec67b0-77a9-4092-846c-7c124c247f27-maas-api-tls\") pod \"2fec67b0-77a9-4092-846c-7c124c247f27\" (UID: \"2fec67b0-77a9-4092-846c-7c124c247f27\") "
Jul 29 01:10:21.032433 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.032410    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fec67b0-77a9-4092-846c-7c124c247f27-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2fec67b0-77a9-4092-846c-7c124c247f27" (UID: "2fec67b0-77a9-4092-846c-7c124c247f27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:10:21.034247 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.034215    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fec67b0-77a9-4092-846c-7c124c247f27-kube-api-access-p4hqw" (OuterVolumeSpecName: "kube-api-access-p4hqw") pod "2fec67b0-77a9-4092-846c-7c124c247f27" (UID: "2fec67b0-77a9-4092-846c-7c124c247f27"). InnerVolumeSpecName "kube-api-access-p4hqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:10:21.034247 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.034215    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fec67b0-77a9-4092-846c-7c124c247f27-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2fec67b0-77a9-4092-846c-7c124c247f27" (UID: "2fec67b0-77a9-4092-846c-7c124c247f27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:10:21.133701 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.133604    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdcda35-70ed-489d-8831-d31b75421bc4-plugins-config-volume\") pod \"1cdcda35-70ed-489d-8831-d31b75421bc4\" (UID: \"1cdcda35-70ed-489d-8831-d31b75421bc4\") "
Jul 29 01:10:21.133701 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.133696    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc46774-9863-45f5-9ad2-6053aed86acd-plugins-config-volume\") pod \"7bc46774-9863-45f5-9ad2-6053aed86acd\" (UID: \"7bc46774-9863-45f5-9ad2-6053aed86acd\") "
Jul 29 01:10:21.133892 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.133732    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f65j\" (UniqueName: \"kubernetes.io/projected/1cdcda35-70ed-489d-8831-d31b75421bc4-kube-api-access-6f65j\") pod \"1cdcda35-70ed-489d-8831-d31b75421bc4\" (UID: \"1cdcda35-70ed-489d-8831-d31b75421bc4\") "
Jul 29 01:10:21.133892 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.133752    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcjz7\" (UniqueName: \"kubernetes.io/projected/7bc46774-9863-45f5-9ad2-6053aed86acd-kube-api-access-hcjz7\") pod \"7bc46774-9863-45f5-9ad2-6053aed86acd\" (UID: \"7bc46774-9863-45f5-9ad2-6053aed86acd\") "
Jul 29 01:10:21.133990 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.133973    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fec67b0-77a9-4092-846c-7c124c247f27-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:10:21.134028 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.133998    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4hqw\" (UniqueName: \"kubernetes.io/projected/2fec67b0-77a9-4092-846c-7c124c247f27-kube-api-access-p4hqw\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:10:21.134028 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.134011    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fec67b0-77a9-4092-846c-7c124c247f27-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:10:21.134091 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.134018    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cdcda35-70ed-489d-8831-d31b75421bc4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cdcda35-70ed-489d-8831-d31b75421bc4" (UID: "1cdcda35-70ed-489d-8831-d31b75421bc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:10:21.134091 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.134070    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc46774-9863-45f5-9ad2-6053aed86acd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bc46774-9863-45f5-9ad2-6053aed86acd" (UID: "7bc46774-9863-45f5-9ad2-6053aed86acd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:10:21.135887 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.135864    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc46774-9863-45f5-9ad2-6053aed86acd-kube-api-access-hcjz7" (OuterVolumeSpecName: "kube-api-access-hcjz7") pod "7bc46774-9863-45f5-9ad2-6053aed86acd" (UID: "7bc46774-9863-45f5-9ad2-6053aed86acd"). InnerVolumeSpecName "kube-api-access-hcjz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:10:21.135954 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.135921    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdcda35-70ed-489d-8831-d31b75421bc4-kube-api-access-6f65j" (OuterVolumeSpecName: "kube-api-access-6f65j") pod "1cdcda35-70ed-489d-8831-d31b75421bc4" (UID: "1cdcda35-70ed-489d-8831-d31b75421bc4"). InnerVolumeSpecName "kube-api-access-6f65j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:10:21.235088 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.235047    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjz7\" (UniqueName: \"kubernetes.io/projected/7bc46774-9863-45f5-9ad2-6053aed86acd-kube-api-access-hcjz7\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:10:21.235088 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.235080    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdcda35-70ed-489d-8831-d31b75421bc4-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:10:21.235088 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.235090    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc46774-9863-45f5-9ad2-6053aed86acd-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:10:21.235088 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.235100    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f65j\" (UniqueName: \"kubernetes.io/projected/1cdcda35-70ed-489d-8831-d31b75421bc4-kube-api-access-6f65j\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:10:21.298515 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.298478    2586 generic.go:356] "Generic (PLEG): container finished" podID="7bc46774-9863-45f5-9ad2-6053aed86acd" containerID="0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae" exitCode=0
Jul 29 01:10:21.298683 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.298524    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"
Jul 29 01:10:21.298683 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.298565    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv" event={"ID":"7bc46774-9863-45f5-9ad2-6053aed86acd","Type":"ContainerDied","Data":"0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae"}
Jul 29 01:10:21.298683 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.298620    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv" event={"ID":"7bc46774-9863-45f5-9ad2-6053aed86acd","Type":"ContainerDied","Data":"620a892dd69255c277eda95156beb8303b062a49870d0c79556e7bf5db0a1210"}
Jul 29 01:10:21.298683 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.298645    2586 scope.go:117] "RemoveContainer" containerID="0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae"
Jul 29 01:10:21.300213 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.300166    2586 generic.go:356] "Generic (PLEG): container finished" podID="2fec67b0-77a9-4092-846c-7c124c247f27" containerID="1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d" exitCode=0
Jul 29 01:10:21.300318 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.300235    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq" event={"ID":"2fec67b0-77a9-4092-846c-7c124c247f27","Type":"ContainerDied","Data":"1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d"}
Jul 29 01:10:21.300318 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.300290    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq" event={"ID":"2fec67b0-77a9-4092-846c-7c124c247f27","Type":"ContainerDied","Data":"6e3e23c003d9b17975b1c3659383c3ad083de30e4833b7a3511054b23b66bca6"}
Jul 29 01:10:21.300318 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.300244    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"
Jul 29 01:10:21.301702 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.301676    2586 generic.go:356] "Generic (PLEG): container finished" podID="1cdcda35-70ed-489d-8831-d31b75421bc4" containerID="14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4" exitCode=0
Jul 29 01:10:21.301817 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.301716    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"
Jul 29 01:10:21.301817 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.301714    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn" event={"ID":"1cdcda35-70ed-489d-8831-d31b75421bc4","Type":"ContainerDied","Data":"14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4"}
Jul 29 01:10:21.301925 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.301853    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn" event={"ID":"1cdcda35-70ed-489d-8831-d31b75421bc4","Type":"ContainerDied","Data":"f6e2c5d1455741d2c499d92d702721e3bc55ae84339bce378fe7b3b217253d2b"}
Jul 29 01:10:21.310340 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.310320    2586 scope.go:117] "RemoveContainer" containerID="0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae"
Jul 29 01:10:21.310645 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:10:21.310628    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae\": container with ID starting with 0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae not found: ID does not exist" containerID="0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae"
Jul 29 01:10:21.310723 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.310652    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae"} err="failed to get container status \"0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae\": rpc error: code = NotFound desc = could not find container \"0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae\": container with ID starting with 0ad7fdb7d70e88fdd809d8f2a67a22178bc352e0fcd4095e8ff2dabd0a5cd9ae not found: ID does not exist"
Jul 29 01:10:21.310723 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.310669    2586 scope.go:117] "RemoveContainer" containerID="1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d"
Jul 29 01:10:21.319800 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.319783    2586 scope.go:117] "RemoveContainer" containerID="1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d"
Jul 29 01:10:21.320057 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:10:21.320039    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d\": container with ID starting with 1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d not found: ID does not exist" containerID="1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d"
Jul 29 01:10:21.320097 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.320065    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d"} err="failed to get container status \"1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d\": rpc error: code = NotFound desc = could not find container \"1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d\": container with ID starting with 1dc3ffa5ad516ea79b9b6b11defcd1a4fd79c79d754ea7f3ed3f5d0f0468652d not found: ID does not exist"
Jul 29 01:10:21.320097 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.320082    2586 scope.go:117] "RemoveContainer" containerID="14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4"
Jul 29 01:10:21.326959 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.326935    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"]
Jul 29 01:10:21.328726 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.328701    2586 scope.go:117] "RemoveContainer" containerID="14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4"
Jul 29 01:10:21.328985 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:10:21.328964    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4\": container with ID starting with 14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4 not found: ID does not exist" containerID="14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4"
Jul 29 01:10:21.329075 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.328992    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4"} err="failed to get container status \"14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4\": rpc error: code = NotFound desc = could not find container \"14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4\": container with ID starting with 14d7fe3aa62882c6b77b873be0762a2de12802703825e986f0df4d33c88335d4 not found: ID does not exist"
Jul 29 01:10:21.330981 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.330958    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9bb5bd21-6b65bf5f48-sx8rv"]
Jul 29 01:10:21.342812 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.342788    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"]
Jul 29 01:10:21.347173 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.347151    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9bb5bd21-567d7745bb-b8xhn"]
Jul 29 01:10:21.358371 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.358331    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"]
Jul 29 01:10:21.360847 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:21.360825    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bb5bd21-865cddf5d8-zdljq"]
Jul 29 01:10:22.221556 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:22.221527    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cdcda35-70ed-489d-8831-d31b75421bc4" path="/var/lib/kubelet/pods/1cdcda35-70ed-489d-8831-d31b75421bc4/volumes"
Jul 29 01:10:22.221922 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:22.221896    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fec67b0-77a9-4092-846c-7c124c247f27" path="/var/lib/kubelet/pods/2fec67b0-77a9-4092-846c-7c124c247f27/volumes"
Jul 29 01:10:22.222218 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:22.222187    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558ef52f-d1c1-49e1-819d-16ac3f82a670" path="/var/lib/kubelet/pods/558ef52f-d1c1-49e1-819d-16ac3f82a670/volumes"
Jul 29 01:10:22.222534 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:22.222522    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc46774-9863-45f5-9ad2-6053aed86acd" path="/var/lib/kubelet/pods/7bc46774-9863-45f5-9ad2-6053aed86acd/volumes"
Jul 29 01:10:23.868711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:23.868680    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:10:28.896251 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:10:28.896189    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:04.706705 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:04.706591    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:05.167748 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:05.167702    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:05.946392 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:05.946344    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:11.468317 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:11.468281    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:12.103483 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:12.103448    2586 scope.go:117] "RemoveContainer" containerID="1480fa8d80fe3c5a16cf51252eea26706a847c27ae065057decf0246686c93d4"
Jul 29 01:11:12.112067 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:12.112047    2586 scope.go:117] "RemoveContainer" containerID="a9a34d7f837a8c14ab380df78c1f1f80ee78da358ebaf82d4fa4c4df072a7a33"
Jul 29 01:11:17.065902 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:17.065861    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:17.231043 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:17.231010    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:17.385288 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:17.381908    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:17.601638 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:17.601606    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:18.668507 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:18.668458    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:18.768246 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:18.768186    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:18.869904 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:18.869834    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:19.582126 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:19.582087    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:19.683382 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:19.683350    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:19.782822 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:19.782790    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:21.163711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:21.163676    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:27.093209 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:27.093174    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:27.192956 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:27.192918    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:27.752522 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:27.752482    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:33.967638 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:33.967600    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:34.070754 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:34.070711    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:34.165470 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:34.165434    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:44.655366 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:44.655328    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:44.977632 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:44.977545    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:11:49.017387 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:11:49.017344    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:05.930578 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:05.930500    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:11.734434 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:11.734400    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:11.835082 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:11.835039    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:12.063846 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:12.063816    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:12.154165 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:12.154136    2586 scope.go:117] "RemoveContainer" containerID="2350c0bd08e4c1c125d3cc0d00d050479492da65e5a4e714760f9ab9e10f2a5e"
Jul 29 01:12:12.162781 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:12.162762    2586 scope.go:117] "RemoveContainer" containerID="3c54476625b0de489a0168f866d11f3c5a96ad17dce1059e711a19be16aefffd"
Jul 29 01:12:12.352653 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:12.352564    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:12.509083 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:12.509055    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:13.570374 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:13.570334    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:13.672007 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:13.671967    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:14.998739 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:14.998700    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:15.078972 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.078932    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:15.219363 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219331    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"]
Jul 29 01:12:15.219710 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219696    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fec67b0-77a9-4092-846c-7c124c247f27" containerName="maas-api"
Jul 29 01:12:15.219710 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219709    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fec67b0-77a9-4092-846c-7c124c247f27" containerName="maas-api"
Jul 29 01:12:15.219823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219717    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc46774-9863-45f5-9ad2-6053aed86acd" containerName="payload-pre-processing"
Jul 29 01:12:15.219823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219724    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc46774-9863-45f5-9ad2-6053aed86acd" containerName="payload-pre-processing"
Jul 29 01:12:15.219823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219732    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdcda35-70ed-489d-8831-d31b75421bc4" containerName="payload-processing"
Jul 29 01:12:15.219823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219737    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdcda35-70ed-489d-8831-d31b75421bc4" containerName="payload-processing"
Jul 29 01:12:15.219823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219744    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558ef52f-d1c1-49e1-819d-16ac3f82a670" containerName="revoke-keys"
Jul 29 01:12:15.219823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219750    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="558ef52f-d1c1-49e1-819d-16ac3f82a670" containerName="revoke-keys"
Jul 29 01:12:15.219823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219808    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fec67b0-77a9-4092-846c-7c124c247f27" containerName="maas-api"
Jul 29 01:12:15.219823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219815    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdcda35-70ed-489d-8831-d31b75421bc4" containerName="payload-processing"
Jul 29 01:12:15.219823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219822    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="558ef52f-d1c1-49e1-819d-16ac3f82a670" containerName="revoke-keys"
Jul 29 01:12:15.220093 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.219828    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc46774-9863-45f5-9ad2-6053aed86acd" containerName="payload-pre-processing"
Jul 29 01:12:15.222945 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.222919    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.225473 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.225452    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9jxwk\""
Jul 29 01:12:15.226083 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.226064    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dda27e-serving-cert\""
Jul 29 01:12:15.226362 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.226347    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 01:12:15.226451 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.226353    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:12:15.234037 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.234015    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"]
Jul 29 01:12:15.241572 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.241548    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b81652-9dcf-4bc6-97db-e873c07dcdcb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.241660 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.241579    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x2cf\" (UniqueName: \"kubernetes.io/projected/21b81652-9dcf-4bc6-97db-e873c07dcdcb-kube-api-access-6x2cf\") pod \"maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.241718 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.241697    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b81652-9dcf-4bc6-97db-e873c07dcdcb-maas-api-tls\") pod \"maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.268080 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.268007    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:15.343009 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.342970    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b81652-9dcf-4bc6-97db-e873c07dcdcb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.343009 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.343010    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x2cf\" (UniqueName: \"kubernetes.io/projected/21b81652-9dcf-4bc6-97db-e873c07dcdcb-kube-api-access-6x2cf\") pod \"maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.343263 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.343069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b81652-9dcf-4bc6-97db-e873c07dcdcb-maas-api-tls\") pod \"maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.343692 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.343671    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b81652-9dcf-4bc6-97db-e873c07dcdcb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.345790 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.345765    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b81652-9dcf-4bc6-97db-e873c07dcdcb-maas-api-tls\") pod \"maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.351938 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.351915    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x2cf\" (UniqueName: \"kubernetes.io/projected/21b81652-9dcf-4bc6-97db-e873c07dcdcb-kube-api-access-6x2cf\") pod \"maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.372791 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.372752    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:15.534603 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.534515    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:15.679178 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.679140    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"]
Jul 29 01:12:15.683215 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:12:15.682846    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b81652_9dcf_4bc6_97db_e873c07dcdcb.slice/crio-f73964894ade46975af69adaf2d4a7bdcfd2e1426e2459849ee3e4fd71707957 WatchSource:0}: Error finding container f73964894ade46975af69adaf2d4a7bdcfd2e1426e2459849ee3e4fd71707957: Status 404 returned error can't find the container with id f73964894ade46975af69adaf2d4a7bdcfd2e1426e2459849ee3e4fd71707957
Jul 29 01:12:15.684252 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.684230    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:12:15.715734 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.715708    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:15.807167 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.807084    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:15.811884 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.811848    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"]
Jul 29 01:12:15.816756 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.816737    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:15.819212 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.819181    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-dda27e\""
Jul 29 01:12:15.827084 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.827061    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"]
Jul 29 01:12:15.847521 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.847484    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ckr\" (UniqueName: \"kubernetes.io/projected/1808a974-6450-4c12-a652-b8626f4e5f18-kube-api-access-r8ckr\") pod \"payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr\" (UID: \"1808a974-6450-4c12-a652-b8626f4e5f18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:15.847681 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.847534    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1808a974-6450-4c12-a652-b8626f4e5f18-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr\" (UID: \"1808a974-6450-4c12-a652-b8626f4e5f18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:15.853105 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.853078    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"]
Jul 29 01:12:15.856654 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.856633    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:15.859285 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.859264    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-dda27e-dockercfg-hhswj\""
Jul 29 01:12:15.866051 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.866025    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"]
Jul 29 01:12:15.948040 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.948005    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8dh\" (UniqueName: \"kubernetes.io/projected/3f52cc17-bdb8-431e-986d-3883cbd03446-kube-api-access-lp8dh\") pod \"payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn\" (UID: \"3f52cc17-bdb8-431e-986d-3883cbd03446\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:15.948253 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.948055    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f52cc17-bdb8-431e-986d-3883cbd03446-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn\" (UID: \"3f52cc17-bdb8-431e-986d-3883cbd03446\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:15.948253 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.948186    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ckr\" (UniqueName: \"kubernetes.io/projected/1808a974-6450-4c12-a652-b8626f4e5f18-kube-api-access-r8ckr\") pod \"payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr\" (UID: \"1808a974-6450-4c12-a652-b8626f4e5f18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:15.948397 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.948254    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1808a974-6450-4c12-a652-b8626f4e5f18-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr\" (UID: \"1808a974-6450-4c12-a652-b8626f4e5f18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:15.948960 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.948936    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1808a974-6450-4c12-a652-b8626f4e5f18-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr\" (UID: \"1808a974-6450-4c12-a652-b8626f4e5f18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:15.956336 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.956312    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ckr\" (UniqueName: \"kubernetes.io/projected/1808a974-6450-4c12-a652-b8626f4e5f18-kube-api-access-r8ckr\") pod \"payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr\" (UID: \"1808a974-6450-4c12-a652-b8626f4e5f18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:15.959900 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:15.959868    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb" event={"ID":"21b81652-9dcf-4bc6-97db-e873c07dcdcb","Type":"ContainerStarted","Data":"f73964894ade46975af69adaf2d4a7bdcfd2e1426e2459849ee3e4fd71707957"}
Jul 29 01:12:16.049211 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.049169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8dh\" (UniqueName: \"kubernetes.io/projected/3f52cc17-bdb8-431e-986d-3883cbd03446-kube-api-access-lp8dh\") pod \"payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn\" (UID: \"3f52cc17-bdb8-431e-986d-3883cbd03446\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:16.049707 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.049271    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f52cc17-bdb8-431e-986d-3883cbd03446-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn\" (UID: \"3f52cc17-bdb8-431e-986d-3883cbd03446\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:16.050052 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.050026    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f52cc17-bdb8-431e-986d-3883cbd03446-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn\" (UID: \"3f52cc17-bdb8-431e-986d-3883cbd03446\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:16.058559 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.058499    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8dh\" (UniqueName: \"kubernetes.io/projected/3f52cc17-bdb8-431e-986d-3883cbd03446-kube-api-access-lp8dh\") pod \"payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn\" (UID: \"3f52cc17-bdb8-431e-986d-3883cbd03446\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:16.129152 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.129116    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:16.169664 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.169627    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:16.333415 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.333386    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"]
Jul 29 01:12:16.335032 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:12:16.335010    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f52cc17_bdb8_431e_986d_3883cbd03446.slice/crio-0cc637baa61a50e0762b8ef847a25377149e8d089a78dc25ee696bcb0e2eeaac WatchSource:0}: Error finding container 0cc637baa61a50e0762b8ef847a25377149e8d089a78dc25ee696bcb0e2eeaac: Status 404 returned error can't find the container with id 0cc637baa61a50e0762b8ef847a25377149e8d089a78dc25ee696bcb0e2eeaac
Jul 29 01:12:16.497519 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.497492    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"]
Jul 29 01:12:16.499100 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:12:16.499068    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1808a974_6450_4c12_a652_b8626f4e5f18.slice/crio-95f333e80128595e0b2f8f5f7849e05a5e3b83ed2ec57ee10b0c5041f37f9ad1 WatchSource:0}: Error finding container 95f333e80128595e0b2f8f5f7849e05a5e3b83ed2ec57ee10b0c5041f37f9ad1: Status 404 returned error can't find the container with id 95f333e80128595e0b2f8f5f7849e05a5e3b83ed2ec57ee10b0c5041f37f9ad1
Jul 29 01:12:16.969620 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.969573    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr" event={"ID":"1808a974-6450-4c12-a652-b8626f4e5f18","Type":"ContainerStarted","Data":"57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea"}
Jul 29 01:12:16.969810 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.969642    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:16.969810 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.969672    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr" event={"ID":"1808a974-6450-4c12-a652-b8626f4e5f18","Type":"ContainerStarted","Data":"95f333e80128595e0b2f8f5f7849e05a5e3b83ed2ec57ee10b0c5041f37f9ad1"}
Jul 29 01:12:16.971327 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.971302    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb" event={"ID":"21b81652-9dcf-4bc6-97db-e873c07dcdcb","Type":"ContainerStarted","Data":"35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e"}
Jul 29 01:12:16.971454 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.971413    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:16.972997 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.972975    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn" event={"ID":"3f52cc17-bdb8-431e-986d-3883cbd03446","Type":"ContainerStarted","Data":"1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94"}
Jul 29 01:12:16.973114 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.973020    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn" event={"ID":"3f52cc17-bdb8-431e-986d-3883cbd03446","Type":"ContainerStarted","Data":"0cc637baa61a50e0762b8ef847a25377149e8d089a78dc25ee696bcb0e2eeaac"}
Jul 29 01:12:16.973114 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.973099    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:16.985680 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:16.985618    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr" podStartSLOduration=1.985595076 podStartE2EDuration="1.985595076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:12:16.985156579 +0000 UTC m=+2167.294084445" watchObservedRunningTime="2026-07-29 01:12:16.985595076 +0000 UTC m=+2167.294522934"
Jul 29 01:12:17.007289 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:17.007238    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn" podStartSLOduration=2.007223471 podStartE2EDuration="2.007223471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:12:17.002473549 +0000 UTC m=+2167.311401403" watchObservedRunningTime="2026-07-29 01:12:17.007223471 +0000 UTC m=+2167.316151333"
Jul 29 01:12:17.025461 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:17.025402    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb" podStartSLOduration=1.5322046230000002 podStartE2EDuration="2.025386822s" totalImagesPullingTime="493.182199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:12:17.023780441 +0000 UTC m=+2167.332708295" watchObservedRunningTime="2026-07-29 01:12:17.025386822 +0000 UTC m=+2167.334314664"
Jul 29 01:12:22.984041 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:22.984009    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:12:27.981403 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:27.981367    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:12:27.981885 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:27.981447    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:12:39.971173 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:39.971136    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:40.274800 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:40.274716    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:40.579847 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:40.579794    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:40.827715 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:40.827680    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:43.101569 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.101506    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:43.197663 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.197635    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:43.202277 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.202248    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"]
Jul 29 01:12:43.206398 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.206378    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.209215 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.209181    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a1744c-serving-cert\""
Jul 29 01:12:43.219686 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.219659    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"]
Jul 29 01:12:43.271450 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.271419    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:43.304663 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.304626    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.304663 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.304664    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rx6\" (UniqueName: \"kubernetes.io/projected/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-kube-api-access-68rx6\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.304859 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.304690    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.405949 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.405864    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.405949 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.405905    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rx6\" (UniqueName: \"kubernetes.io/projected/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-kube-api-access-68rx6\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.405949 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.405930    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.406171 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:12:43.406040    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-serving-cert: secret "maas-api-e2e-inf-b-a1744c-serving-cert" not found
Jul 29 01:12:43.406171 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:12:43.406114    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-maas-api-tls podName:190cb892-c2d8-41fa-a8dc-3bcf967fae1a nodeName:}" failed. No retries permitted until 2026-07-29 01:12:43.906094636 +0000 UTC m=+2194.215022457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-maas-api-tls") pod "maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m" (UID: "190cb892-c2d8-41fa-a8dc-3bcf967fae1a") : secret "maas-api-e2e-inf-b-a1744c-serving-cert" not found
Jul 29 01:12:43.406540 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.406520    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.416882 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.416859    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rx6\" (UniqueName: \"kubernetes.io/projected/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-kube-api-access-68rx6\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.545112 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.545079    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:43.912319 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.912286    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:43.914719 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:43.914696    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:44.048730 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.048693    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:44.118417 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.118383    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:44.155025 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.154994    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:12:44.243111 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.243069    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"]
Jul 29 01:12:44.246468 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.246446    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:12:44.251356 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.251334    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a1744c-dockercfg-92j64\""
Jul 29 01:12:44.251864 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.251842    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a1744c\""
Jul 29 01:12:44.267076 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.267044    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"]
Jul 29 01:12:44.269606 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.269584    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"]
Jul 29 01:12:44.270768 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:12:44.270740    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190cb892_c2d8_41fa_a8dc_3bcf967fae1a.slice/crio-e8c71b8b3ad79a4d6671c7bda80e679b20e1fe536c84037b4cbf41b500c16a71 WatchSource:0}: Error finding container e8c71b8b3ad79a4d6671c7bda80e679b20e1fe536c84037b4cbf41b500c16a71: Status 404 returned error can't find the container with id e8c71b8b3ad79a4d6671c7bda80e679b20e1fe536c84037b4cbf41b500c16a71
Jul 29 01:12:44.328156 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.328117    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"]
Jul 29 01:12:44.331978 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.331956    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:12:44.344155 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.344127    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"]
Jul 29 01:12:44.417476 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.417441    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536cd337-b02a-4ee8-95b8-47f99ce13d6e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd\" (UID: \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:12:44.417476 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.417474    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnzx\" (UniqueName: \"kubernetes.io/projected/536cd337-b02a-4ee8-95b8-47f99ce13d6e-kube-api-access-5qnzx\") pod \"payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd\" (UID: \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:12:44.417704 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.417507    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5adfde3-a164-4978-976f-03e577fac192-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw\" (UID: \"b5adfde3-a164-4978-976f-03e577fac192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:12:44.417704 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.417646    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdjd\" (UniqueName: \"kubernetes.io/projected/b5adfde3-a164-4978-976f-03e577fac192-kube-api-access-xtdjd\") pod \"payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw\" (UID: \"b5adfde3-a164-4978-976f-03e577fac192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:12:44.518645 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.518552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536cd337-b02a-4ee8-95b8-47f99ce13d6e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd\" (UID: \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:12:44.518645 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.518592    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnzx\" (UniqueName: \"kubernetes.io/projected/536cd337-b02a-4ee8-95b8-47f99ce13d6e-kube-api-access-5qnzx\") pod \"payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd\" (UID: \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:12:44.518645 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.518641    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5adfde3-a164-4978-976f-03e577fac192-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw\" (UID: \"b5adfde3-a164-4978-976f-03e577fac192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:12:44.518883 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.518736    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdjd\" (UniqueName: \"kubernetes.io/projected/b5adfde3-a164-4978-976f-03e577fac192-kube-api-access-xtdjd\") pod \"payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw\" (UID: \"b5adfde3-a164-4978-976f-03e577fac192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:12:44.519406 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.519386    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536cd337-b02a-4ee8-95b8-47f99ce13d6e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd\" (UID: \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:12:44.519406 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.519396    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5adfde3-a164-4978-976f-03e577fac192-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw\" (UID: \"b5adfde3-a164-4978-976f-03e577fac192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:12:44.527294 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.527259    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdjd\" (UniqueName: \"kubernetes.io/projected/b5adfde3-a164-4978-976f-03e577fac192-kube-api-access-xtdjd\") pod \"payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw\" (UID: \"b5adfde3-a164-4978-976f-03e577fac192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:12:44.530536 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.530507    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnzx\" (UniqueName: \"kubernetes.io/projected/536cd337-b02a-4ee8-95b8-47f99ce13d6e-kube-api-access-5qnzx\") pod \"payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd\" (UID: \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:12:44.559354 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.559314    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:12:44.644799 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.644742    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:12:44.708645 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.708616    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"]
Jul 29 01:12:44.710737 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:12:44.710696    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5adfde3_a164_4978_976f_03e577fac192.slice/crio-08bf37a5d0e44207b8b64d30acace8c2b205b9f6ddda93b2ccffe0700b50e8a6 WatchSource:0}: Error finding container 08bf37a5d0e44207b8b64d30acace8c2b205b9f6ddda93b2ccffe0700b50e8a6: Status 404 returned error can't find the container with id 08bf37a5d0e44207b8b64d30acace8c2b205b9f6ddda93b2ccffe0700b50e8a6
Jul 29 01:12:44.823676 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:44.823648    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"]
Jul 29 01:12:44.825135 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:12:44.825110    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536cd337_b02a_4ee8_95b8_47f99ce13d6e.slice/crio-7890b51f673a3e7ad62c17bd7f5aa3ce4cf45b90dfab82d9ec515ef246d8f978 WatchSource:0}: Error finding container 7890b51f673a3e7ad62c17bd7f5aa3ce4cf45b90dfab82d9ec515ef246d8f978: Status 404 returned error can't find the container with id 7890b51f673a3e7ad62c17bd7f5aa3ce4cf45b90dfab82d9ec515ef246d8f978
Jul 29 01:12:45.156132 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.156048    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw" event={"ID":"b5adfde3-a164-4978-976f-03e577fac192","Type":"ContainerStarted","Data":"93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d"}
Jul 29 01:12:45.156132 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.156115    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:12:45.156132 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.156139    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw" event={"ID":"b5adfde3-a164-4978-976f-03e577fac192","Type":"ContainerStarted","Data":"08bf37a5d0e44207b8b64d30acace8c2b205b9f6ddda93b2ccffe0700b50e8a6"}
Jul 29 01:12:45.158461 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.158425    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m" event={"ID":"190cb892-c2d8-41fa-a8dc-3bcf967fae1a","Type":"ContainerStarted","Data":"602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b"}
Jul 29 01:12:45.158599 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.158486    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m" event={"ID":"190cb892-c2d8-41fa-a8dc-3bcf967fae1a","Type":"ContainerStarted","Data":"e8c71b8b3ad79a4d6671c7bda80e679b20e1fe536c84037b4cbf41b500c16a71"}
Jul 29 01:12:45.158778 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.158748    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:45.160412 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.160378    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd" event={"ID":"536cd337-b02a-4ee8-95b8-47f99ce13d6e","Type":"ContainerStarted","Data":"ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc"}
Jul 29 01:12:45.160673 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.160657    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:12:45.160827 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.160806    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd" event={"ID":"536cd337-b02a-4ee8-95b8-47f99ce13d6e","Type":"ContainerStarted","Data":"7890b51f673a3e7ad62c17bd7f5aa3ce4cf45b90dfab82d9ec515ef246d8f978"}
Jul 29 01:12:45.174879 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.174833    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw" podStartSLOduration=1.174820273 podStartE2EDuration="1.174820273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:12:45.173349894 +0000 UTC m=+2195.482277736" watchObservedRunningTime="2026-07-29 01:12:45.174820273 +0000 UTC m=+2195.483748109"
Jul 29 01:12:45.191525 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.191481    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd" podStartSLOduration=1.191467392 podStartE2EDuration="1.191467392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:12:45.190872159 +0000 UTC m=+2195.499800001" watchObservedRunningTime="2026-07-29 01:12:45.191467392 +0000 UTC m=+2195.500395234"
Jul 29 01:12:45.209237 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:45.209151    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m" podStartSLOduration=1.868708046 podStartE2EDuration="2.209130866s" totalImagesPullingTime="340.42282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:12:45.209031518 +0000 UTC m=+2195.517959360" watchObservedRunningTime="2026-07-29 01:12:45.209130866 +0000 UTC m=+2195.518058709"
Jul 29 01:12:51.173768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:51.173738    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:12:56.169862 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:56.169817    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw" podUID="b5adfde3-a164-4978-976f-03e577fac192" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.81:9004: connect: connection refused"
Jul 29 01:12:56.170456 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:12:56.170306    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:13:06.169313 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:13:06.169274    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:13:08.099827 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:13:08.099792    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:13:10.732411 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:13:10.732379    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:13:33.619185 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:13:33.619109    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:12.196347 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:12.196311    2586 scope.go:117] "RemoveContainer" containerID="b0582744c1c2993109b3fb8525b374a16c5baa19bc123cb6560aeeb7bf5ee8a7"
Jul 29 01:14:27.985397 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:27.985358    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:28.946640 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:28.946595    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:29.050233 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.050178    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"]
Jul 29 01:14:29.053949 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.053926    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"
Jul 29 01:14:29.057100 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.057076    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:14:29.066571 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.066544    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"]
Jul 29 01:14:29.181849 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.181811    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349b0418-6c75-4cc7-b58d-697e9e96abdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl\" (UID: \"349b0418-6c75-4cc7-b58d-697e9e96abdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"
Jul 29 01:14:29.283398 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.283303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349b0418-6c75-4cc7-b58d-697e9e96abdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl\" (UID: \"349b0418-6c75-4cc7-b58d-697e9e96abdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"
Jul 29 01:14:29.283964 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.283945    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349b0418-6c75-4cc7-b58d-697e9e96abdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl\" (UID: \"349b0418-6c75-4cc7-b58d-697e9e96abdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"
Jul 29 01:14:29.366824 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.366785    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"
Jul 29 01:14:29.511579 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.511552    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"]
Jul 29 01:14:29.513170 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:14:29.513142    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349b0418_6c75_4cc7_b58d_697e9e96abdf.slice/crio-d7db6cb73eee46f7e5a004f2dabb533052af31b78d3281682bea6e25934a3c19 WatchSource:0}: Error finding container d7db6cb73eee46f7e5a004f2dabb533052af31b78d3281682bea6e25934a3c19: Status 404 returned error can't find the container with id d7db6cb73eee46f7e5a004f2dabb533052af31b78d3281682bea6e25934a3c19
Jul 29 01:14:29.779171 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:29.779138    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl" event={"ID":"349b0418-6c75-4cc7-b58d-697e9e96abdf","Type":"ContainerStarted","Data":"d7db6cb73eee46f7e5a004f2dabb533052af31b78d3281682bea6e25934a3c19"}
Jul 29 01:14:30.787524 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:30.787491    2586 generic.go:356] "Generic (PLEG): container finished" podID="349b0418-6c75-4cc7-b58d-697e9e96abdf" containerID="f11af363536fa578ee9b9be6a33777c9e258d4e0b9e670326ffb929c9249e23d" exitCode=0
Jul 29 01:14:30.787930 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:30.787554    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl" event={"ID":"349b0418-6c75-4cc7-b58d-697e9e96abdf","Type":"ContainerDied","Data":"f11af363536fa578ee9b9be6a33777c9e258d4e0b9e670326ffb929c9249e23d"}
Jul 29 01:14:31.939945 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:31.939914    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"
Jul 29 01:14:32.111712 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:32.111669    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349b0418-6c75-4cc7-b58d-697e9e96abdf-maas-api-service-ca\") pod \"349b0418-6c75-4cc7-b58d-697e9e96abdf\" (UID: \"349b0418-6c75-4cc7-b58d-697e9e96abdf\") "
Jul 29 01:14:32.112022 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:32.111999    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349b0418-6c75-4cc7-b58d-697e9e96abdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "349b0418-6c75-4cc7-b58d-697e9e96abdf" (UID: "349b0418-6c75-4cc7-b58d-697e9e96abdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:14:32.212963 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:32.212919    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349b0418-6c75-4cc7-b58d-697e9e96abdf-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:32.800850 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:32.800816    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl" event={"ID":"349b0418-6c75-4cc7-b58d-697e9e96abdf","Type":"ContainerDied","Data":"d7db6cb73eee46f7e5a004f2dabb533052af31b78d3281682bea6e25934a3c19"}
Jul 29 01:14:32.800850 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:32.800843    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"
Jul 29 01:14:32.801050 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:32.800863    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7db6cb73eee46f7e5a004f2dabb533052af31b78d3281682bea6e25934a3c19"
Jul 29 01:14:39.437279 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.437240    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"]
Jul 29 01:14:39.437967 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.437505    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb" podUID="21b81652-9dcf-4bc6-97db-e873c07dcdcb" containerName="maas-api" containerID="cri-o://35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e" gracePeriod=30
Jul 29 01:14:39.459586 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.459522    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:39.487493 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.487459    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"]
Jul 29 01:14:39.487751 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.487726    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn" podUID="3f52cc17-bdb8-431e-986d-3883cbd03446" containerName="payload-processing" containerID="cri-o://1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94" gracePeriod=30
Jul 29 01:14:39.492573 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.492270    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"]
Jul 29 01:14:39.492573 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.492538    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr" podUID="1808a974-6450-4c12-a652-b8626f4e5f18" containerName="payload-pre-processing" containerID="cri-o://57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea" gracePeriod=30
Jul 29 01:14:39.526575 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.526548    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:39.628112 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.628066    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:39.639082 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.639044    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"]
Jul 29 01:14:39.643549 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.643424    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dda27e-2phvl"]
Jul 29 01:14:39.684457 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.684349    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:39.762919 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.762869    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:14:39.778172 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.778139    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x2cf\" (UniqueName: \"kubernetes.io/projected/21b81652-9dcf-4bc6-97db-e873c07dcdcb-kube-api-access-6x2cf\") pod \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") "
Jul 29 01:14:39.778370 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.778292    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b81652-9dcf-4bc6-97db-e873c07dcdcb-openshift-service-ca\") pod \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") "
Jul 29 01:14:39.778370 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.778324    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b81652-9dcf-4bc6-97db-e873c07dcdcb-maas-api-tls\") pod \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\" (UID: \"21b81652-9dcf-4bc6-97db-e873c07dcdcb\") "
Jul 29 01:14:39.779859 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.779831    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b81652-9dcf-4bc6-97db-e873c07dcdcb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21b81652-9dcf-4bc6-97db-e873c07dcdcb" (UID: "21b81652-9dcf-4bc6-97db-e873c07dcdcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:14:39.781649 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.781302    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b81652-9dcf-4bc6-97db-e873c07dcdcb-kube-api-access-6x2cf" (OuterVolumeSpecName: "kube-api-access-6x2cf") pod "21b81652-9dcf-4bc6-97db-e873c07dcdcb" (UID: "21b81652-9dcf-4bc6-97db-e873c07dcdcb"). InnerVolumeSpecName "kube-api-access-6x2cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:14:39.781649 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.781318    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b81652-9dcf-4bc6-97db-e873c07dcdcb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21b81652-9dcf-4bc6-97db-e873c07dcdcb" (UID: "21b81652-9dcf-4bc6-97db-e873c07dcdcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:14:39.793966 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.793936    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:14:39.798393 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.798369    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:14:39.841155 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.841119    2586 generic.go:356] "Generic (PLEG): container finished" podID="3f52cc17-bdb8-431e-986d-3883cbd03446" containerID="1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94" exitCode=0
Jul 29 01:14:39.841382 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.841167    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"
Jul 29 01:14:39.841382 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.841215    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn" event={"ID":"3f52cc17-bdb8-431e-986d-3883cbd03446","Type":"ContainerDied","Data":"1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94"}
Jul 29 01:14:39.841382 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.841280    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn" event={"ID":"3f52cc17-bdb8-431e-986d-3883cbd03446","Type":"ContainerDied","Data":"0cc637baa61a50e0762b8ef847a25377149e8d089a78dc25ee696bcb0e2eeaac"}
Jul 29 01:14:39.841382 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.841311    2586 scope.go:117] "RemoveContainer" containerID="1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94"
Jul 29 01:14:39.842916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.842892    2586 generic.go:356] "Generic (PLEG): container finished" podID="1808a974-6450-4c12-a652-b8626f4e5f18" containerID="57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea" exitCode=0
Jul 29 01:14:39.843021 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.842952    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"
Jul 29 01:14:39.843021 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.842960    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr" event={"ID":"1808a974-6450-4c12-a652-b8626f4e5f18","Type":"ContainerDied","Data":"57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea"}
Jul 29 01:14:39.843404 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.843380    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr" event={"ID":"1808a974-6450-4c12-a652-b8626f4e5f18","Type":"ContainerDied","Data":"95f333e80128595e0b2f8f5f7849e05a5e3b83ed2ec57ee10b0c5041f37f9ad1"}
Jul 29 01:14:39.844647 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.844615    2586 generic.go:356] "Generic (PLEG): container finished" podID="21b81652-9dcf-4bc6-97db-e873c07dcdcb" containerID="35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e" exitCode=0
Jul 29 01:14:39.844761 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.844647    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb" event={"ID":"21b81652-9dcf-4bc6-97db-e873c07dcdcb","Type":"ContainerDied","Data":"35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e"}
Jul 29 01:14:39.844761 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.844673    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"
Jul 29 01:14:39.844761 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.844698    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb" event={"ID":"21b81652-9dcf-4bc6-97db-e873c07dcdcb","Type":"ContainerDied","Data":"f73964894ade46975af69adaf2d4a7bdcfd2e1426e2459849ee3e4fd71707957"}
Jul 29 01:14:39.851802 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.851784    2586 scope.go:117] "RemoveContainer" containerID="1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94"
Jul 29 01:14:39.852123 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:14:39.852099    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94\": container with ID starting with 1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94 not found: ID does not exist" containerID="1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94"
Jul 29 01:14:39.852279 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.852130    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94"} err="failed to get container status \"1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94\": rpc error: code = NotFound desc = could not find container \"1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94\": container with ID starting with 1deb326d154aa9eff856551a43d174247c7185e3de86105a51990e9ad7f14a94 not found: ID does not exist"
Jul 29 01:14:39.852279 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.852148    2586 scope.go:117] "RemoveContainer" containerID="57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea"
Jul 29 01:14:39.864720 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.864703    2586 scope.go:117] "RemoveContainer" containerID="57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea"
Jul 29 01:14:39.865016 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:14:39.864996    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea\": container with ID starting with 57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea not found: ID does not exist" containerID="57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea"
Jul 29 01:14:39.865080 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.865024    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea"} err="failed to get container status \"57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea\": rpc error: code = NotFound desc = could not find container \"57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea\": container with ID starting with 57f9d7ebb0390157c7ba90269b664a001a9d5ba8228f0609e2ff4319b24e8bea not found: ID does not exist"
Jul 29 01:14:39.865080 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.865040    2586 scope.go:117] "RemoveContainer" containerID="35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e"
Jul 29 01:14:39.874113 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.874094    2586 scope.go:117] "RemoveContainer" containerID="35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e"
Jul 29 01:14:39.874421 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:14:39.874401    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e\": container with ID starting with 35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e not found: ID does not exist" containerID="35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e"
Jul 29 01:14:39.874500 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.874431    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e"} err="failed to get container status \"35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e\": rpc error: code = NotFound desc = could not find container \"35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e\": container with ID starting with 35f450684ffaca64c713956dea149707a6708a8712931b1343b9361bf5c2892e not found: ID does not exist"
Jul 29 01:14:39.874669 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.874650    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"]
Jul 29 01:14:39.877904 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.877882    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dda27e-696685d7f6-knzlb"]
Jul 29 01:14:39.879368 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.879348    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8ckr\" (UniqueName: \"kubernetes.io/projected/1808a974-6450-4c12-a652-b8626f4e5f18-kube-api-access-r8ckr\") pod \"1808a974-6450-4c12-a652-b8626f4e5f18\" (UID: \"1808a974-6450-4c12-a652-b8626f4e5f18\") "
Jul 29 01:14:39.879464 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.879402    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f52cc17-bdb8-431e-986d-3883cbd03446-plugins-config-volume\") pod \"3f52cc17-bdb8-431e-986d-3883cbd03446\" (UID: \"3f52cc17-bdb8-431e-986d-3883cbd03446\") "
Jul 29 01:14:39.879464 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.879444    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1808a974-6450-4c12-a652-b8626f4e5f18-plugins-config-volume\") pod \"1808a974-6450-4c12-a652-b8626f4e5f18\" (UID: \"1808a974-6450-4c12-a652-b8626f4e5f18\") "
Jul 29 01:14:39.879583 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.879536    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp8dh\" (UniqueName: \"kubernetes.io/projected/3f52cc17-bdb8-431e-986d-3883cbd03446-kube-api-access-lp8dh\") pod \"3f52cc17-bdb8-431e-986d-3883cbd03446\" (UID: \"3f52cc17-bdb8-431e-986d-3883cbd03446\") "
Jul 29 01:14:39.879777 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.879761    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x2cf\" (UniqueName: \"kubernetes.io/projected/21b81652-9dcf-4bc6-97db-e873c07dcdcb-kube-api-access-6x2cf\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:39.879836 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.879782    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b81652-9dcf-4bc6-97db-e873c07dcdcb-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:39.879836 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.879796    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b81652-9dcf-4bc6-97db-e873c07dcdcb-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:39.879836 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.879805    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f52cc17-bdb8-431e-986d-3883cbd03446-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f52cc17-bdb8-431e-986d-3883cbd03446" (UID: "3f52cc17-bdb8-431e-986d-3883cbd03446"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:14:39.879937 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.879839    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1808a974-6450-4c12-a652-b8626f4e5f18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1808a974-6450-4c12-a652-b8626f4e5f18" (UID: "1808a974-6450-4c12-a652-b8626f4e5f18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:14:39.881563 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.881545    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1808a974-6450-4c12-a652-b8626f4e5f18-kube-api-access-r8ckr" (OuterVolumeSpecName: "kube-api-access-r8ckr") pod "1808a974-6450-4c12-a652-b8626f4e5f18" (UID: "1808a974-6450-4c12-a652-b8626f4e5f18"). InnerVolumeSpecName "kube-api-access-r8ckr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:14:39.881624 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.881571    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f52cc17-bdb8-431e-986d-3883cbd03446-kube-api-access-lp8dh" (OuterVolumeSpecName: "kube-api-access-lp8dh") pod "3f52cc17-bdb8-431e-986d-3883cbd03446" (UID: "3f52cc17-bdb8-431e-986d-3883cbd03446"). InnerVolumeSpecName "kube-api-access-lp8dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:14:39.980763 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.980669    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8ckr\" (UniqueName: \"kubernetes.io/projected/1808a974-6450-4c12-a652-b8626f4e5f18-kube-api-access-r8ckr\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:39.980763 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.980705    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f52cc17-bdb8-431e-986d-3883cbd03446-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:39.980763 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.980715    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1808a974-6450-4c12-a652-b8626f4e5f18-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:39.980763 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.980723    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8dh\" (UniqueName: \"kubernetes.io/projected/3f52cc17-bdb8-431e-986d-3883cbd03446-kube-api-access-lp8dh\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:39.997926 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:39.997892    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:40.164583 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:40.164551    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"]
Jul 29 01:14:40.168354 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:40.168313    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dda27e-5566f49c8d-h9fpn"]
Jul 29 01:14:40.178732 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:40.178699    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"]
Jul 29 01:14:40.182167 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:40.182143    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dda27e-65d5948cbd-5q4zr"]
Jul 29 01:14:40.223265 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:40.223230    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1808a974-6450-4c12-a652-b8626f4e5f18" path="/var/lib/kubelet/pods/1808a974-6450-4c12-a652-b8626f4e5f18/volumes"
Jul 29 01:14:40.223663 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:40.223648    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b81652-9dcf-4bc6-97db-e873c07dcdcb" path="/var/lib/kubelet/pods/21b81652-9dcf-4bc6-97db-e873c07dcdcb/volumes"
Jul 29 01:14:40.223946 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:40.223934    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="349b0418-6c75-4cc7-b58d-697e9e96abdf" path="/var/lib/kubelet/pods/349b0418-6c75-4cc7-b58d-697e9e96abdf/volumes"
Jul 29 01:14:40.224282 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:40.224269    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f52cc17-bdb8-431e-986d-3883cbd03446" path="/var/lib/kubelet/pods/3f52cc17-bdb8-431e-986d-3883cbd03446/volumes"
Jul 29 01:14:46.063008 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.062980    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:46.163364 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.163331    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:46.423626 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.423543    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:46.581015 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.580973    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:46.688047 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.687951    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"]
Jul 29 01:14:46.688481 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688459    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b81652-9dcf-4bc6-97db-e873c07dcdcb" containerName="maas-api"
Jul 29 01:14:46.688594 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688483    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b81652-9dcf-4bc6-97db-e873c07dcdcb" containerName="maas-api"
Jul 29 01:14:46.688594 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688504    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f52cc17-bdb8-431e-986d-3883cbd03446" containerName="payload-processing"
Jul 29 01:14:46.688594 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688511    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f52cc17-bdb8-431e-986d-3883cbd03446" containerName="payload-processing"
Jul 29 01:14:46.688594 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688545    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1808a974-6450-4c12-a652-b8626f4e5f18" containerName="payload-pre-processing"
Jul 29 01:14:46.688594 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688555    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1808a974-6450-4c12-a652-b8626f4e5f18" containerName="payload-pre-processing"
Jul 29 01:14:46.688594 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688569    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="349b0418-6c75-4cc7-b58d-697e9e96abdf" containerName="revoke-keys"
Jul 29 01:14:46.688594 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688577    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="349b0418-6c75-4cc7-b58d-697e9e96abdf" containerName="revoke-keys"
Jul 29 01:14:46.688926 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688681    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b81652-9dcf-4bc6-97db-e873c07dcdcb" containerName="maas-api"
Jul 29 01:14:46.688926 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688694    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1808a974-6450-4c12-a652-b8626f4e5f18" containerName="payload-pre-processing"
Jul 29 01:14:46.688926 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688705    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="349b0418-6c75-4cc7-b58d-697e9e96abdf" containerName="revoke-keys"
Jul 29 01:14:46.688926 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.688718    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f52cc17-bdb8-431e-986d-3883cbd03446" containerName="payload-processing"
Jul 29 01:14:46.692017 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.691991    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"
Jul 29 01:14:46.694682 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.694660    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:14:46.699932 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.699898    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"]
Jul 29 01:14:46.730160 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.730123    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8c1618-1045-4624-bc34-bbfcbbee15eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a1744c-7k744\" (UID: \"cc8c1618-1045-4624-bc34-bbfcbbee15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"
Jul 29 01:14:46.830741 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.830697    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8c1618-1045-4624-bc34-bbfcbbee15eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a1744c-7k744\" (UID: \"cc8c1618-1045-4624-bc34-bbfcbbee15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"
Jul 29 01:14:46.831612 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:46.831588    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8c1618-1045-4624-bc34-bbfcbbee15eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a1744c-7k744\" (UID: \"cc8c1618-1045-4624-bc34-bbfcbbee15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"
Jul 29 01:14:47.006039 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:47.005948    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"
Jul 29 01:14:47.180508 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:47.180480    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"]
Jul 29 01:14:47.182984 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:14:47.182945    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8c1618_1045_4624_bc34_bbfcbbee15eb.slice/crio-4e63a332816aaeea07095041d86db134368515c27d88ee09095d3fa3c709f95f WatchSource:0}: Error finding container 4e63a332816aaeea07095041d86db134368515c27d88ee09095d3fa3c709f95f: Status 404 returned error can't find the container with id 4e63a332816aaeea07095041d86db134368515c27d88ee09095d3fa3c709f95f
Jul 29 01:14:47.901646 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:47.901573    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744" event={"ID":"cc8c1618-1045-4624-bc34-bbfcbbee15eb","Type":"ContainerStarted","Data":"07ef990ef52dbb94cab87052dd5565ea0c16e764ccf8cd4b8ee1f888e088c271"}
Jul 29 01:14:47.901863 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:47.901713    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744" event={"ID":"cc8c1618-1045-4624-bc34-bbfcbbee15eb","Type":"ContainerStarted","Data":"4e63a332816aaeea07095041d86db134368515c27d88ee09095d3fa3c709f95f"}
Jul 29 01:14:47.915897 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:47.915837    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744" podStartSLOduration=1.915819828 podStartE2EDuration="1.915819828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:14:47.915657171 +0000 UTC m=+2318.224585015" watchObservedRunningTime="2026-07-29 01:14:47.915819828 +0000 UTC m=+2318.224747670"
Jul 29 01:14:48.908733 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:48.908696    2586 generic.go:356] "Generic (PLEG): container finished" podID="cc8c1618-1045-4624-bc34-bbfcbbee15eb" containerID="07ef990ef52dbb94cab87052dd5565ea0c16e764ccf8cd4b8ee1f888e088c271" exitCode=0
Jul 29 01:14:48.909142 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:48.908764    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744" event={"ID":"cc8c1618-1045-4624-bc34-bbfcbbee15eb","Type":"ContainerDied","Data":"07ef990ef52dbb94cab87052dd5565ea0c16e764ccf8cd4b8ee1f888e088c271"}
Jul 29 01:14:50.055544 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:50.055513    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"
Jul 29 01:14:50.155129 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:50.155085    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8c1618-1045-4624-bc34-bbfcbbee15eb-maas-api-service-ca\") pod \"cc8c1618-1045-4624-bc34-bbfcbbee15eb\" (UID: \"cc8c1618-1045-4624-bc34-bbfcbbee15eb\") "
Jul 29 01:14:50.155583 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:50.155553    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8c1618-1045-4624-bc34-bbfcbbee15eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc8c1618-1045-4624-bc34-bbfcbbee15eb" (UID: "cc8c1618-1045-4624-bc34-bbfcbbee15eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:14:50.256085 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:50.256054    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8c1618-1045-4624-bc34-bbfcbbee15eb-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:50.921719 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:50.921691    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"
Jul 29 01:14:50.921881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:50.921690    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744" event={"ID":"cc8c1618-1045-4624-bc34-bbfcbbee15eb","Type":"ContainerDied","Data":"4e63a332816aaeea07095041d86db134368515c27d88ee09095d3fa3c709f95f"}
Jul 29 01:14:50.921881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:50.921817    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e63a332816aaeea07095041d86db134368515c27d88ee09095d3fa3c709f95f"
Jul 29 01:14:57.023870 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.023830    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"]
Jul 29 01:14:57.024737 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.024656    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m" podUID="190cb892-c2d8-41fa-a8dc-3bcf967fae1a" containerName="maas-api" containerID="cri-o://602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b" gracePeriod=30
Jul 29 01:14:57.048969 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.048937    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:57.076404 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.076374    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"]
Jul 29 01:14:57.076663 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.076613    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd" podUID="536cd337-b02a-4ee8-95b8-47f99ce13d6e" containerName="payload-processing" containerID="cri-o://ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc" gracePeriod=30
Jul 29 01:14:57.082079 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.082051    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"]
Jul 29 01:14:57.082344 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.082291    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw" podUID="b5adfde3-a164-4978-976f-03e577fac192" containerName="payload-pre-processing" containerID="cri-o://93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d" gracePeriod=30
Jul 29 01:14:57.117456 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.117424    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:57.203666 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.203627    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"]
Jul 29 01:14:57.203809 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.203686    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a1744c-7k744"]
Jul 29 01:14:57.208797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.208760    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:57.263509 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.262572    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:57.355533 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.355499    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:14:57.381672 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.381649    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:14:57.392486 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.392465    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:14:57.414824 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.414794    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536cd337-b02a-4ee8-95b8-47f99ce13d6e-plugins-config-volume\") pod \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\" (UID: \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\") "
Jul 29 01:14:57.414976 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.414871    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5adfde3-a164-4978-976f-03e577fac192-plugins-config-volume\") pod \"b5adfde3-a164-4978-976f-03e577fac192\" (UID: \"b5adfde3-a164-4978-976f-03e577fac192\") "
Jul 29 01:14:57.415066 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.414980    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtdjd\" (UniqueName: \"kubernetes.io/projected/b5adfde3-a164-4978-976f-03e577fac192-kube-api-access-xtdjd\") pod \"b5adfde3-a164-4978-976f-03e577fac192\" (UID: \"b5adfde3-a164-4978-976f-03e577fac192\") "
Jul 29 01:14:57.415066 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.415043    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68rx6\" (UniqueName: \"kubernetes.io/projected/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-kube-api-access-68rx6\") pod \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") "
Jul 29 01:14:57.415187 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.415071    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qnzx\" (UniqueName: \"kubernetes.io/projected/536cd337-b02a-4ee8-95b8-47f99ce13d6e-kube-api-access-5qnzx\") pod \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\" (UID: \"536cd337-b02a-4ee8-95b8-47f99ce13d6e\") "
Jul 29 01:14:57.415187 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.415132    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-openshift-service-ca\") pod \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") "
Jul 29 01:14:57.415187 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.415157    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-maas-api-tls\") pod \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\" (UID: \"190cb892-c2d8-41fa-a8dc-3bcf967fae1a\") "
Jul 29 01:14:57.415355 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.415262    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536cd337-b02a-4ee8-95b8-47f99ce13d6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "536cd337-b02a-4ee8-95b8-47f99ce13d6e" (UID: "536cd337-b02a-4ee8-95b8-47f99ce13d6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:14:57.415355 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.415275    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5adfde3-a164-4978-976f-03e577fac192-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5adfde3-a164-4978-976f-03e577fac192" (UID: "b5adfde3-a164-4978-976f-03e577fac192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:14:57.415548 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.415522    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5adfde3-a164-4978-976f-03e577fac192-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:57.415619 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.415552    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536cd337-b02a-4ee8-95b8-47f99ce13d6e-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:57.415995 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.415972    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "190cb892-c2d8-41fa-a8dc-3bcf967fae1a" (UID: "190cb892-c2d8-41fa-a8dc-3bcf967fae1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:14:57.418307 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.418278    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-kube-api-access-68rx6" (OuterVolumeSpecName: "kube-api-access-68rx6") pod "190cb892-c2d8-41fa-a8dc-3bcf967fae1a" (UID: "190cb892-c2d8-41fa-a8dc-3bcf967fae1a"). InnerVolumeSpecName "kube-api-access-68rx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:14:57.418415 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.418314    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "190cb892-c2d8-41fa-a8dc-3bcf967fae1a" (UID: "190cb892-c2d8-41fa-a8dc-3bcf967fae1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:14:57.418415 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.418354    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5adfde3-a164-4978-976f-03e577fac192-kube-api-access-xtdjd" (OuterVolumeSpecName: "kube-api-access-xtdjd") pod "b5adfde3-a164-4978-976f-03e577fac192" (UID: "b5adfde3-a164-4978-976f-03e577fac192"). InnerVolumeSpecName "kube-api-access-xtdjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:14:57.418514 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.418446    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536cd337-b02a-4ee8-95b8-47f99ce13d6e-kube-api-access-5qnzx" (OuterVolumeSpecName: "kube-api-access-5qnzx") pod "536cd337-b02a-4ee8-95b8-47f99ce13d6e" (UID: "536cd337-b02a-4ee8-95b8-47f99ce13d6e"). InnerVolumeSpecName "kube-api-access-5qnzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:14:57.516926 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.516898    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtdjd\" (UniqueName: \"kubernetes.io/projected/b5adfde3-a164-4978-976f-03e577fac192-kube-api-access-xtdjd\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:57.516926 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.516923    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rx6\" (UniqueName: \"kubernetes.io/projected/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-kube-api-access-68rx6\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:57.516926 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.516932    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnzx\" (UniqueName: \"kubernetes.io/projected/536cd337-b02a-4ee8-95b8-47f99ce13d6e-kube-api-access-5qnzx\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:57.517168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.516945    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:57.517168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.516973    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cb892-c2d8-41fa-a8dc-3bcf967fae1a-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:14:57.568123 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.568092    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:14:57.964579 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.964478    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5adfde3-a164-4978-976f-03e577fac192" containerID="93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d" exitCode=0
Jul 29 01:14:57.964579 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.964528    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"
Jul 29 01:14:57.964855 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.964567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw" event={"ID":"b5adfde3-a164-4978-976f-03e577fac192","Type":"ContainerDied","Data":"93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d"}
Jul 29 01:14:57.964855 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.964664    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw" event={"ID":"b5adfde3-a164-4978-976f-03e577fac192","Type":"ContainerDied","Data":"08bf37a5d0e44207b8b64d30acace8c2b205b9f6ddda93b2ccffe0700b50e8a6"}
Jul 29 01:14:57.964855 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.964699    2586 scope.go:117] "RemoveContainer" containerID="93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d"
Jul 29 01:14:57.966023 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.965999    2586 generic.go:356] "Generic (PLEG): container finished" podID="190cb892-c2d8-41fa-a8dc-3bcf967fae1a" containerID="602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b" exitCode=0
Jul 29 01:14:57.966167 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.966048    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"
Jul 29 01:14:57.966167 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.966091    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m" event={"ID":"190cb892-c2d8-41fa-a8dc-3bcf967fae1a","Type":"ContainerDied","Data":"602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b"}
Jul 29 01:14:57.966167 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.966138    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m" event={"ID":"190cb892-c2d8-41fa-a8dc-3bcf967fae1a","Type":"ContainerDied","Data":"e8c71b8b3ad79a4d6671c7bda80e679b20e1fe536c84037b4cbf41b500c16a71"}
Jul 29 01:14:57.967591 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.967564    2586 generic.go:356] "Generic (PLEG): container finished" podID="536cd337-b02a-4ee8-95b8-47f99ce13d6e" containerID="ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc" exitCode=0
Jul 29 01:14:57.967698 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.967617    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"
Jul 29 01:14:57.967698 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.967625    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd" event={"ID":"536cd337-b02a-4ee8-95b8-47f99ce13d6e","Type":"ContainerDied","Data":"ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc"}
Jul 29 01:14:57.967698 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.967663    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd" event={"ID":"536cd337-b02a-4ee8-95b8-47f99ce13d6e","Type":"ContainerDied","Data":"7890b51f673a3e7ad62c17bd7f5aa3ce4cf45b90dfab82d9ec515ef246d8f978"}
Jul 29 01:14:57.974391 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.974374    2586 scope.go:117] "RemoveContainer" containerID="93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d"
Jul 29 01:14:57.974697 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:14:57.974675    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d\": container with ID starting with 93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d not found: ID does not exist" containerID="93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d"
Jul 29 01:14:57.974788 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.974707    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d"} err="failed to get container status \"93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d\": rpc error: code = NotFound desc = could not find container \"93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d\": container with ID starting with 93471bd2a628f73765d1f2edf26679e6c07b4093ceb713114dba93f0a69c432d not found: ID does not exist"
Jul 29 01:14:57.974788 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.974730    2586 scope.go:117] "RemoveContainer" containerID="602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b"
Jul 29 01:14:57.985162 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.985145    2586 scope.go:117] "RemoveContainer" containerID="602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b"
Jul 29 01:14:57.985461 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:14:57.985439    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b\": container with ID starting with 602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b not found: ID does not exist" containerID="602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b"
Jul 29 01:14:57.985532 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.985468    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b"} err="failed to get container status \"602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b\": rpc error: code = NotFound desc = could not find container \"602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b\": container with ID starting with 602419765d9ba2c06a444982b92ca02c45d9c0fb0cd7fe4923cac3d7f858fb8b not found: ID does not exist"
Jul 29 01:14:57.985532 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.985485    2586 scope.go:117] "RemoveContainer" containerID="ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc"
Jul 29 01:14:57.992665 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.992643    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"]
Jul 29 01:14:57.993905 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.993878    2586 scope.go:117] "RemoveContainer" containerID="ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc"
Jul 29 01:14:57.994319 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:14:57.994293    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc\": container with ID starting with ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc not found: ID does not exist" containerID="ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc"
Jul 29 01:14:57.994412 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.994329    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc"} err="failed to get container status \"ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc\": rpc error: code = NotFound desc = could not find container \"ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc\": container with ID starting with ec15a5950e5df6b1218667ff15aec759e8fea32cb3e98b45d7557794f674f8fc not found: ID does not exist"
Jul 29 01:14:57.996867 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:57.996846    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a1744c-55c5d7f6b8-xcj4m"]
Jul 29 01:14:58.007661 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:58.007636    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"]
Jul 29 01:14:58.014166 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:58.014144    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a1744c-75b497cc96-rncpd"]
Jul 29 01:14:58.024900 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:58.024870    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"]
Jul 29 01:14:58.029294 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:58.029273    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a1744c-7d7f66484-9dfqw"]
Jul 29 01:14:58.222595 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:58.222504    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="190cb892-c2d8-41fa-a8dc-3bcf967fae1a" path="/var/lib/kubelet/pods/190cb892-c2d8-41fa-a8dc-3bcf967fae1a/volumes"
Jul 29 01:14:58.222927 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:58.222911    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536cd337-b02a-4ee8-95b8-47f99ce13d6e" path="/var/lib/kubelet/pods/536cd337-b02a-4ee8-95b8-47f99ce13d6e/volumes"
Jul 29 01:14:58.223249 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:58.223237    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5adfde3-a164-4978-976f-03e577fac192" path="/var/lib/kubelet/pods/b5adfde3-a164-4978-976f-03e577fac192/volumes"
Jul 29 01:14:58.223521 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:14:58.223511    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc8c1618-1045-4624-bc34-bbfcbbee15eb" path="/var/lib/kubelet/pods/cc8c1618-1045-4624-bc34-bbfcbbee15eb/volumes"
Jul 29 01:15:00.133216 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133161    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv"]
Jul 29 01:15:00.133641 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133561    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190cb892-c2d8-41fa-a8dc-3bcf967fae1a" containerName="maas-api"
Jul 29 01:15:00.133641 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133572    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="190cb892-c2d8-41fa-a8dc-3bcf967fae1a" containerName="maas-api"
Jul 29 01:15:00.133641 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133599    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5adfde3-a164-4978-976f-03e577fac192" containerName="payload-pre-processing"
Jul 29 01:15:00.133641 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133604    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5adfde3-a164-4978-976f-03e577fac192" containerName="payload-pre-processing"
Jul 29 01:15:00.133641 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133611    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536cd337-b02a-4ee8-95b8-47f99ce13d6e" containerName="payload-processing"
Jul 29 01:15:00.133641 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133617    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="536cd337-b02a-4ee8-95b8-47f99ce13d6e" containerName="payload-processing"
Jul 29 01:15:00.133641 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133624    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8c1618-1045-4624-bc34-bbfcbbee15eb" containerName="revoke-keys"
Jul 29 01:15:00.133641 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133630    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8c1618-1045-4624-bc34-bbfcbbee15eb" containerName="revoke-keys"
Jul 29 01:15:00.133924 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133684    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="536cd337-b02a-4ee8-95b8-47f99ce13d6e" containerName="payload-processing"
Jul 29 01:15:00.133924 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133694    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="190cb892-c2d8-41fa-a8dc-3bcf967fae1a" containerName="maas-api"
Jul 29 01:15:00.133924 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133700    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8c1618-1045-4624-bc34-bbfcbbee15eb" containerName="revoke-keys"
Jul 29 01:15:00.133924 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.133705    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5adfde3-a164-4978-976f-03e577fac192" containerName="payload-pre-processing"
Jul 29 01:15:00.138497 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.138476    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv"
Jul 29 01:15:00.141243 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.141188    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:15:00.152536 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.145024    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv"]
Jul 29 01:15:00.158434 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.158397    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv"
Jul 29 01:15:00.303112 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.303083    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv"]
Jul 29 01:15:00.305359 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:15:00.305331    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78805b92_92b6_4255_938a_7a6228005a83.slice/crio-eec775f4d6a327f050ce77c62e900801576a3bad0e290e7ecdc32698eac9d6d2 WatchSource:0}: Error finding container eec775f4d6a327f050ce77c62e900801576a3bad0e290e7ecdc32698eac9d6d2: Status 404 returned error can't find the container with id eec775f4d6a327f050ce77c62e900801576a3bad0e290e7ecdc32698eac9d6d2
Jul 29 01:15:00.991826 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.991782    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv" event={"ID":"78805b92-92b6-4255-938a-7a6228005a83","Type":"ContainerStarted","Data":"081f18b38dfcc54622e8c8ef61c0439f690d7d98898e1fb7c241d0fc9972cde2"}
Jul 29 01:15:00.992039 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:00.991866    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv" event={"ID":"78805b92-92b6-4255-938a-7a6228005a83","Type":"ContainerStarted","Data":"eec775f4d6a327f050ce77c62e900801576a3bad0e290e7ecdc32698eac9d6d2"}
Jul 29 01:15:01.007346 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:01.007293    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv" podStartSLOduration=1.007279714 podStartE2EDuration="1.007279714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:01.005989319 +0000 UTC m=+2331.314917161" watchObservedRunningTime="2026-07-29 01:15:01.007279714 +0000 UTC m=+2331.316207598"
Jul 29 01:15:02.000167 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:02.000075    2586 generic.go:356] "Generic (PLEG): container finished" podID="78805b92-92b6-4255-938a-7a6228005a83" containerID="081f18b38dfcc54622e8c8ef61c0439f690d7d98898e1fb7c241d0fc9972cde2" exitCode=0
Jul 29 01:15:02.000566 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:02.000160    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv" event={"ID":"78805b92-92b6-4255-938a-7a6228005a83","Type":"ContainerDied","Data":"081f18b38dfcc54622e8c8ef61c0439f690d7d98898e1fb7c241d0fc9972cde2"}
Jul 29 01:15:03.141073 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:03.141046    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv"
Jul 29 01:15:03.550003 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:03.549967    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:03.614499 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:03.614460    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:03.804167 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:03.804064    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:04.013028 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:04.012996    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv"
Jul 29 01:15:04.013028 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:04.013019    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754795-9x4vv" event={"ID":"78805b92-92b6-4255-938a-7a6228005a83","Type":"ContainerDied","Data":"eec775f4d6a327f050ce77c62e900801576a3bad0e290e7ecdc32698eac9d6d2"}
Jul 29 01:15:04.013267 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:04.013068    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eec775f4d6a327f050ce77c62e900801576a3bad0e290e7ecdc32698eac9d6d2"
Jul 29 01:15:04.038798 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:04.038746    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:04.355967 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:04.355932    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:04.684569 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:04.684489    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:04.935528 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:04.935436    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:07.913461 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:07.913422    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:08.017771 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.017736    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:08.019176 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.019149    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"]
Jul 29 01:15:08.019615 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.019597    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78805b92-92b6-4255-938a-7a6228005a83" containerName="cleanup"
Jul 29 01:15:08.019697 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.019617    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="78805b92-92b6-4255-938a-7a6228005a83" containerName="cleanup"
Jul 29 01:15:08.019747 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.019712    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="78805b92-92b6-4255-938a-7a6228005a83" containerName="cleanup"
Jul 29 01:15:08.023023 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.023001    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.025557 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.025533    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4650f4-serving-cert\""
Jul 29 01:15:08.025706 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.025543    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:15:08.025706 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.025687    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 01:15:08.026697 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.026680    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9jxwk\""
Jul 29 01:15:08.031610 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.031580    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"]
Jul 29 01:15:08.083829 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.083797    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:08.110107 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.110068    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ace133-9720-4260-a4b1-04518c8f0599-openshift-service-ca\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.110309 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.110118    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ace133-9720-4260-a4b1-04518c8f0599-maas-api-tls\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.110309 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.110216    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbgx\" (UniqueName: \"kubernetes.io/projected/26ace133-9720-4260-a4b1-04518c8f0599-kube-api-access-pwbgx\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.211265 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.211163    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ace133-9720-4260-a4b1-04518c8f0599-openshift-service-ca\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.211265 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.211226    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ace133-9720-4260-a4b1-04518c8f0599-maas-api-tls\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.211464 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.211283    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbgx\" (UniqueName: \"kubernetes.io/projected/26ace133-9720-4260-a4b1-04518c8f0599-kube-api-access-pwbgx\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.211464 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:15:08.211386    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-serving-cert: secret "maas-api-e2e-api-a-4650f4-serving-cert" not found
Jul 29 01:15:08.211535 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:15:08.211502    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ace133-9720-4260-a4b1-04518c8f0599-maas-api-tls podName:26ace133-9720-4260-a4b1-04518c8f0599 nodeName:}" failed. No retries permitted until 2026-07-29 01:15:08.711475377 +0000 UTC m=+2339.020403217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26ace133-9720-4260-a4b1-04518c8f0599-maas-api-tls") pod "maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4" (UID: "26ace133-9720-4260-a4b1-04518c8f0599") : secret "maas-api-e2e-api-a-4650f4-serving-cert" not found
Jul 29 01:15:08.211952 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.211934    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ace133-9720-4260-a4b1-04518c8f0599-openshift-service-ca\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.222300 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.222267    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbgx\" (UniqueName: \"kubernetes.io/projected/26ace133-9720-4260-a4b1-04518c8f0599-kube-api-access-pwbgx\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.563103 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.563059    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:08.675927 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.675887    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:08.715920 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.715888    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ace133-9720-4260-a4b1-04518c8f0599-maas-api-tls\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.718466 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.718440    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ace133-9720-4260-a4b1-04518c8f0599-maas-api-tls\") pod \"maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:08.780944 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.780915    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:08.782477 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.782451    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"]
Jul 29 01:15:08.786291 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.786273    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:08.788888 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.788865    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4650f4-dockercfg-5dr4k\""
Jul 29 01:15:08.788993 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.788865    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4650f4\""
Jul 29 01:15:08.795276 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.795253    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"]
Jul 29 01:15:08.798811 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.798790    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"]
Jul 29 01:15:08.802243 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.802187    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:08.813540 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.813482    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"]
Jul 29 01:15:08.816987 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.816963    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2gm2\" (UniqueName: \"kubernetes.io/projected/44a022fb-0c10-40d1-8b21-7462bbef7c62-kube-api-access-j2gm2\") pod \"payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk\" (UID: \"44a022fb-0c10-40d1-8b21-7462bbef7c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:08.817096 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.817078    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a022fb-0c10-40d1-8b21-7462bbef7c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk\" (UID: \"44a022fb-0c10-40d1-8b21-7462bbef7c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:08.917844 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.917800    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a022fb-0c10-40d1-8b21-7462bbef7c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk\" (UID: \"44a022fb-0c10-40d1-8b21-7462bbef7c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:08.918262 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.917852    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974tn\" (UniqueName: \"kubernetes.io/projected/7e685140-ec23-4d07-8a46-981e420f673f-kube-api-access-974tn\") pod \"payload-processing-e2e-api-a-4650f4-c66b4578-hcxts\" (UID: \"7e685140-ec23-4d07-8a46-981e420f673f\") " pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:08.918262 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.917913    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2gm2\" (UniqueName: \"kubernetes.io/projected/44a022fb-0c10-40d1-8b21-7462bbef7c62-kube-api-access-j2gm2\") pod \"payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk\" (UID: \"44a022fb-0c10-40d1-8b21-7462bbef7c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:08.918262 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.917943    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e685140-ec23-4d07-8a46-981e420f673f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4650f4-c66b4578-hcxts\" (UID: \"7e685140-ec23-4d07-8a46-981e420f673f\") " pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:08.918571 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.918553    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a022fb-0c10-40d1-8b21-7462bbef7c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk\" (UID: \"44a022fb-0c10-40d1-8b21-7462bbef7c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:08.925633 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.925603    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2gm2\" (UniqueName: \"kubernetes.io/projected/44a022fb-0c10-40d1-8b21-7462bbef7c62-kube-api-access-j2gm2\") pod \"payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk\" (UID: \"44a022fb-0c10-40d1-8b21-7462bbef7c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:08.935441 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:08.935412    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:09.018739 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:09.018703    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e685140-ec23-4d07-8a46-981e420f673f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4650f4-c66b4578-hcxts\" (UID: \"7e685140-ec23-4d07-8a46-981e420f673f\") " pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:09.018917 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:09.018813    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974tn\" (UniqueName: \"kubernetes.io/projected/7e685140-ec23-4d07-8a46-981e420f673f-kube-api-access-974tn\") pod \"payload-processing-e2e-api-a-4650f4-c66b4578-hcxts\" (UID: \"7e685140-ec23-4d07-8a46-981e420f673f\") " pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:09.019703 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:09.019659    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e685140-ec23-4d07-8a46-981e420f673f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4650f4-c66b4578-hcxts\" (UID: \"7e685140-ec23-4d07-8a46-981e420f673f\") " pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:09.034844 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:09.034810    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-974tn\" (UniqueName: \"kubernetes.io/projected/7e685140-ec23-4d07-8a46-981e420f673f-kube-api-access-974tn\") pod \"payload-processing-e2e-api-a-4650f4-c66b4578-hcxts\" (UID: \"7e685140-ec23-4d07-8a46-981e420f673f\") " pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:09.095464 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:09.095438    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"]
Jul 29 01:15:09.098172 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:15:09.098139    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ace133_9720_4260_a4b1_04518c8f0599.slice/crio-6cebe17c30e7ebd63f129c45be997402c3a21aecf94b79a5d521f392c01faa2a WatchSource:0}: Error finding container 6cebe17c30e7ebd63f129c45be997402c3a21aecf94b79a5d521f392c01faa2a: Status 404 returned error can't find the container with id 6cebe17c30e7ebd63f129c45be997402c3a21aecf94b79a5d521f392c01faa2a
Jul 29 01:15:09.098809 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:09.098791    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:09.115907 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:09.115851    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:09.241230 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:09.241183    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"]
Jul 29 01:15:09.241987 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:15:09.241947    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a022fb_0c10_40d1_8b21_7462bbef7c62.slice/crio-a3c725e6c695c92996caaa67659798412fb9e9775670151b7cc2e9fb0ecb3ab3 WatchSource:0}: Error finding container a3c725e6c695c92996caaa67659798412fb9e9775670151b7cc2e9fb0ecb3ab3: Status 404 returned error can't find the container with id a3c725e6c695c92996caaa67659798412fb9e9775670151b7cc2e9fb0ecb3ab3
Jul 29 01:15:09.272541 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:09.272504    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"]
Jul 29 01:15:09.283353 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:15:09.283326    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e685140_ec23_4d07_8a46_981e420f673f.slice/crio-ab4d5d521fffbf973a4ec17fa12ff555ad154a7a0d5c8f4dac8e84ded86c3710 WatchSource:0}: Error finding container ab4d5d521fffbf973a4ec17fa12ff555ad154a7a0d5c8f4dac8e84ded86c3710: Status 404 returned error can't find the container with id ab4d5d521fffbf973a4ec17fa12ff555ad154a7a0d5c8f4dac8e84ded86c3710
Jul 29 01:15:10.053373 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.053267    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk" event={"ID":"44a022fb-0c10-40d1-8b21-7462bbef7c62","Type":"ContainerStarted","Data":"52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779"}
Jul 29 01:15:10.053373 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.053333    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:10.053373 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.053357    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk" event={"ID":"44a022fb-0c10-40d1-8b21-7462bbef7c62","Type":"ContainerStarted","Data":"a3c725e6c695c92996caaa67659798412fb9e9775670151b7cc2e9fb0ecb3ab3"}
Jul 29 01:15:10.054979 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.054951    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts" event={"ID":"7e685140-ec23-4d07-8a46-981e420f673f","Type":"ContainerStarted","Data":"ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4"}
Jul 29 01:15:10.055124 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.055003    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts" event={"ID":"7e685140-ec23-4d07-8a46-981e420f673f","Type":"ContainerStarted","Data":"ab4d5d521fffbf973a4ec17fa12ff555ad154a7a0d5c8f4dac8e84ded86c3710"}
Jul 29 01:15:10.055124 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.055060    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:10.056809 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.056785    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4" event={"ID":"26ace133-9720-4260-a4b1-04518c8f0599","Type":"ContainerStarted","Data":"4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826"}
Jul 29 01:15:10.056910 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.056830    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4" event={"ID":"26ace133-9720-4260-a4b1-04518c8f0599","Type":"ContainerStarted","Data":"6cebe17c30e7ebd63f129c45be997402c3a21aecf94b79a5d521f392c01faa2a"}
Jul 29 01:15:10.057089 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.057041    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:10.080991 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.080923    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk" podStartSLOduration=2.080901134 podStartE2EDuration="2.080901134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:10.067979544 +0000 UTC m=+2340.376907388" watchObservedRunningTime="2026-07-29 01:15:10.080901134 +0000 UTC m=+2340.389828981"
Jul 29 01:15:10.084508 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.084455    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts" podStartSLOduration=2.084436649 podStartE2EDuration="2.084436649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:10.082803656 +0000 UTC m=+2340.391731499" watchObservedRunningTime="2026-07-29 01:15:10.084436649 +0000 UTC m=+2340.393364493"
Jul 29 01:15:10.100966 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:10.100905    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4" podStartSLOduration=2.537589931 podStartE2EDuration="3.100884312s" totalImagesPullingTime="563.294381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:10.099332191 +0000 UTC m=+2340.408260043" watchObservedRunningTime="2026-07-29 01:15:10.100884312 +0000 UTC m=+2340.409812157"
Jul 29 01:15:12.218649 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:12.218614    2586 scope.go:117] "RemoveContainer" containerID="2002795bf28a0bd90b77acc357c36ea5fcba967beaecd6b47bebaa104932bdc2"
Jul 29 01:15:16.068626 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:16.068592    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:15:21.064933 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:21.064899    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:15:21.065389 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:21.064972    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:15:32.804907 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:32.804870    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:33.272173 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:33.272130    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:33.577642 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:33.577608    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:33.915776 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:33.915677    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:36.025793 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.025753    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:36.116098 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.116058    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:36.118996 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.118962    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"]
Jul 29 01:15:36.122843 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.122815    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.125938 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.125914    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d5d9a9-serving-cert\""
Jul 29 01:15:36.134899 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.134862    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"]
Jul 29 01:15:36.191922 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.191880    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:36.272463 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.272423    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b8312f-0e9c-47dd-8102-813f4663f1e6-maas-api-tls\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.272669 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.272551    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b8312f-0e9c-47dd-8102-813f4663f1e6-openshift-service-ca\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.272669 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.272590    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svnwg\" (UniqueName: \"kubernetes.io/projected/14b8312f-0e9c-47dd-8102-813f4663f1e6-kube-api-access-svnwg\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.373770 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.373724    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b8312f-0e9c-47dd-8102-813f4663f1e6-maas-api-tls\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.373995 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:15:36.373872    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-serving-cert: secret "maas-api-e2e-api-b-d5d9a9-serving-cert" not found
Jul 29 01:15:36.373995 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.373940    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b8312f-0e9c-47dd-8102-813f4663f1e6-openshift-service-ca\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.373995 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:15:36.373949    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b8312f-0e9c-47dd-8102-813f4663f1e6-maas-api-tls podName:14b8312f-0e9c-47dd-8102-813f4663f1e6 nodeName:}" failed. No retries permitted until 2026-07-29 01:15:36.87392983 +0000 UTC m=+2367.182857650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14b8312f-0e9c-47dd-8102-813f4663f1e6-maas-api-tls") pod "maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv" (UID: "14b8312f-0e9c-47dd-8102-813f4663f1e6") : secret "maas-api-e2e-api-b-d5d9a9-serving-cert" not found
Jul 29 01:15:36.373995 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.373987    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svnwg\" (UniqueName: \"kubernetes.io/projected/14b8312f-0e9c-47dd-8102-813f4663f1e6-kube-api-access-svnwg\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.374626 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.374607    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b8312f-0e9c-47dd-8102-813f4663f1e6-openshift-service-ca\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.386856 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.386820    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svnwg\" (UniqueName: \"kubernetes.io/projected/14b8312f-0e9c-47dd-8102-813f4663f1e6-kube-api-access-svnwg\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.523804 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.523760    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:36.679159 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.679067    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:36.780539 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.780489    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:15:36.786644 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.786607    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"]
Jul 29 01:15:36.790271 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.790249    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:15:36.793721 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.793691    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d5d9a9\""
Jul 29 01:15:36.793864 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.793806    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d5d9a9-dockercfg-hmnf7\""
Jul 29 01:15:36.803118 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.803090    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"]
Jul 29 01:15:36.815552 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.815524    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"]
Jul 29 01:15:36.819044 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.819026    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:36.831309 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.831281    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"]
Jul 29 01:15:36.879552 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.879508    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ea29dd-d1ee-45df-8155-e610a153e714-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc\" (UID: \"61ea29dd-d1ee-45df-8155-e610a153e714\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:15:36.879759 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.879681    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfsp\" (UniqueName: \"kubernetes.io/projected/61ea29dd-d1ee-45df-8155-e610a153e714-kube-api-access-7jfsp\") pod \"payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc\" (UID: \"61ea29dd-d1ee-45df-8155-e610a153e714\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:15:36.879842 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.879765    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b8312f-0e9c-47dd-8102-813f4663f1e6-maas-api-tls\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.882504 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.882476    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b8312f-0e9c-47dd-8102-813f4663f1e6-maas-api-tls\") pod \"maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:36.981278 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.981148    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfsp\" (UniqueName: \"kubernetes.io/projected/61ea29dd-d1ee-45df-8155-e610a153e714-kube-api-access-7jfsp\") pod \"payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc\" (UID: \"61ea29dd-d1ee-45df-8155-e610a153e714\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:15:36.981460 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.981328    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69r6b\" (UniqueName: \"kubernetes.io/projected/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-kube-api-access-69r6b\") pod \"payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59\" (UID: \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\") " pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:36.981545 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.981518    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59\" (UID: \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\") " pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:36.981701 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.981666    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ea29dd-d1ee-45df-8155-e610a153e714-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc\" (UID: \"61ea29dd-d1ee-45df-8155-e610a153e714\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:15:36.982608 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.982582    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ea29dd-d1ee-45df-8155-e610a153e714-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc\" (UID: \"61ea29dd-d1ee-45df-8155-e610a153e714\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:15:36.990922 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:36.990892    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfsp\" (UniqueName: \"kubernetes.io/projected/61ea29dd-d1ee-45df-8155-e610a153e714-kube-api-access-7jfsp\") pod \"payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc\" (UID: \"61ea29dd-d1ee-45df-8155-e610a153e714\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:15:37.035732 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.035690    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:37.082211 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.082164    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59\" (UID: \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\") " pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:37.082386 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.082354    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69r6b\" (UniqueName: \"kubernetes.io/projected/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-kube-api-access-69r6b\") pod \"payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59\" (UID: \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\") " pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:37.082961 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.082942    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59\" (UID: \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\") " pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:37.090792 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.090738    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69r6b\" (UniqueName: \"kubernetes.io/projected/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-kube-api-access-69r6b\") pod \"payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59\" (UID: \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\") " pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:37.102665 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.102402    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:15:37.132588 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.132533    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:37.213005 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.212952    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"]
Jul 29 01:15:37.229665 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.229588    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv" event={"ID":"14b8312f-0e9c-47dd-8102-813f4663f1e6","Type":"ContainerStarted","Data":"494a21628e6429460610f5ff920cdb64e14403cb8a0f10cca752236799adbf0b"}
Jul 29 01:15:37.293232 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.293179    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"]
Jul 29 01:15:37.294750 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:15:37.294719    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ea29dd_d1ee_45df_8155_e610a153e714.slice/crio-537e78516c54fc7a566651407c412672e6aa0ed6adf0bf2c111810178933c86b WatchSource:0}: Error finding container 537e78516c54fc7a566651407c412672e6aa0ed6adf0bf2c111810178933c86b: Status 404 returned error can't find the container with id 537e78516c54fc7a566651407c412672e6aa0ed6adf0bf2c111810178933c86b
Jul 29 01:15:37.344147 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:37.344116    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"]
Jul 29 01:15:37.344793 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:15:37.344768    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a0b745_5b3a_4376_95f5_3ffc6e5b3553.slice/crio-15bc6e54fd61912ac3b5978ed67547645bce7f4011d64cc316f8bf001f0c7223 WatchSource:0}: Error finding container 15bc6e54fd61912ac3b5978ed67547645bce7f4011d64cc316f8bf001f0c7223: Status 404 returned error can't find the container with id 15bc6e54fd61912ac3b5978ed67547645bce7f4011d64cc316f8bf001f0c7223
Jul 29 01:15:38.238515 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.238474    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc" event={"ID":"61ea29dd-d1ee-45df-8155-e610a153e714","Type":"ContainerStarted","Data":"3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549"}
Jul 29 01:15:38.239035 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.238535    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc" event={"ID":"61ea29dd-d1ee-45df-8155-e610a153e714","Type":"ContainerStarted","Data":"537e78516c54fc7a566651407c412672e6aa0ed6adf0bf2c111810178933c86b"}
Jul 29 01:15:38.239035 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.238667    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:15:38.240240 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.240213    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv" event={"ID":"14b8312f-0e9c-47dd-8102-813f4663f1e6","Type":"ContainerStarted","Data":"40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448"}
Jul 29 01:15:38.240373 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.240267    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:38.241862 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.241837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59" event={"ID":"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553","Type":"ContainerStarted","Data":"5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04"}
Jul 29 01:15:38.241977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.241872    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59" event={"ID":"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553","Type":"ContainerStarted","Data":"15bc6e54fd61912ac3b5978ed67547645bce7f4011d64cc316f8bf001f0c7223"}
Jul 29 01:15:38.241977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.241898    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:38.254993 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.254929    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc" podStartSLOduration=2.254907608 podStartE2EDuration="2.254907608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:38.253889592 +0000 UTC m=+2368.562817438" watchObservedRunningTime="2026-07-29 01:15:38.254907608 +0000 UTC m=+2368.563835451"
Jul 29 01:15:38.270754 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.270691    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv" podStartSLOduration=1.727877675 podStartE2EDuration="2.270675829s" totalImagesPullingTime="542.798154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:38.269943095 +0000 UTC m=+2368.578870961" watchObservedRunningTime="2026-07-29 01:15:38.270675829 +0000 UTC m=+2368.579603671"
Jul 29 01:15:38.287277 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:38.287182    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59" podStartSLOduration=2.287163266 podStartE2EDuration="2.287163266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:38.285106554 +0000 UTC m=+2368.594034389" watchObservedRunningTime="2026-07-29 01:15:38.287163266 +0000 UTC m=+2368.596091107"
Jul 29 01:15:44.254099 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:44.254064    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:15:49.250080 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:49.250044    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:15:49.250552 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:15:49.250257    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:16:00.985593 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:00.985557    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:12.266731 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:12.266694    2586 scope.go:117] "RemoveContainer" containerID="34ed6ae6cc7c2b494fd46366551c3b8e2152d578b4b812837222d4e2b5642668"
Jul 29 01:16:14.438286 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:14.438249    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:19.388937 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.388895    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"]
Jul 29 01:16:19.389822 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.389765    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4" podUID="26ace133-9720-4260-a4b1-04518c8f0599" containerName="maas-api" containerID="cri-o://4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826" gracePeriod=30
Jul 29 01:16:19.412258 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.412180    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:19.440640 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.440603    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"]
Jul 29 01:16:19.440953 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.440855    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts" podUID="7e685140-ec23-4d07-8a46-981e420f673f" containerName="payload-processing" containerID="cri-o://ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4" gracePeriod=30
Jul 29 01:16:19.460220 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.460168    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"]
Jul 29 01:16:19.460888 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.460837    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk" podUID="44a022fb-0c10-40d1-8b21-7462bbef7c62" containerName="payload-pre-processing" containerID="cri-o://52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779" gracePeriod=30
Jul 29 01:16:19.478054 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.477081    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:19.538424 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.538394    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:19.601743 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.601635    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:19.670587 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.670480    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:16:19.714714 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.714680    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:16:19.754757 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.754732    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:16:19.766995 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.766961    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwbgx\" (UniqueName: \"kubernetes.io/projected/26ace133-9720-4260-a4b1-04518c8f0599-kube-api-access-pwbgx\") pod \"26ace133-9720-4260-a4b1-04518c8f0599\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") "
Jul 29 01:16:19.766995 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.767000    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ace133-9720-4260-a4b1-04518c8f0599-maas-api-tls\") pod \"26ace133-9720-4260-a4b1-04518c8f0599\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") "
Jul 29 01:16:19.767314 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.767090    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ace133-9720-4260-a4b1-04518c8f0599-openshift-service-ca\") pod \"26ace133-9720-4260-a4b1-04518c8f0599\" (UID: \"26ace133-9720-4260-a4b1-04518c8f0599\") "
Jul 29 01:16:19.767522 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.767497    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ace133-9720-4260-a4b1-04518c8f0599-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26ace133-9720-4260-a4b1-04518c8f0599" (UID: "26ace133-9720-4260-a4b1-04518c8f0599"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:16:19.769229 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.769180    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ace133-9720-4260-a4b1-04518c8f0599-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26ace133-9720-4260-a4b1-04518c8f0599" (UID: "26ace133-9720-4260-a4b1-04518c8f0599"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:16:19.769339 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.769252    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ace133-9720-4260-a4b1-04518c8f0599-kube-api-access-pwbgx" (OuterVolumeSpecName: "kube-api-access-pwbgx") pod "26ace133-9720-4260-a4b1-04518c8f0599" (UID: "26ace133-9720-4260-a4b1-04518c8f0599"). InnerVolumeSpecName "kube-api-access-pwbgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:16:19.867835 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.867788    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2gm2\" (UniqueName: \"kubernetes.io/projected/44a022fb-0c10-40d1-8b21-7462bbef7c62-kube-api-access-j2gm2\") pod \"44a022fb-0c10-40d1-8b21-7462bbef7c62\" (UID: \"44a022fb-0c10-40d1-8b21-7462bbef7c62\") "
Jul 29 01:16:19.868016 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.867883    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a022fb-0c10-40d1-8b21-7462bbef7c62-plugins-config-volume\") pod \"44a022fb-0c10-40d1-8b21-7462bbef7c62\" (UID: \"44a022fb-0c10-40d1-8b21-7462bbef7c62\") "
Jul 29 01:16:19.868016 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.867910    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-974tn\" (UniqueName: \"kubernetes.io/projected/7e685140-ec23-4d07-8a46-981e420f673f-kube-api-access-974tn\") pod \"7e685140-ec23-4d07-8a46-981e420f673f\" (UID: \"7e685140-ec23-4d07-8a46-981e420f673f\") "
Jul 29 01:16:19.868016 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.867962    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e685140-ec23-4d07-8a46-981e420f673f-plugins-config-volume\") pod \"7e685140-ec23-4d07-8a46-981e420f673f\" (UID: \"7e685140-ec23-4d07-8a46-981e420f673f\") "
Jul 29 01:16:19.868212 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.868177    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ace133-9720-4260-a4b1-04518c8f0599-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:19.868288 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.868221    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwbgx\" (UniqueName: \"kubernetes.io/projected/26ace133-9720-4260-a4b1-04518c8f0599-kube-api-access-pwbgx\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:19.868288 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.868232    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ace133-9720-4260-a4b1-04518c8f0599-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:19.868411 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.868343    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a022fb-0c10-40d1-8b21-7462bbef7c62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44a022fb-0c10-40d1-8b21-7462bbef7c62" (UID: "44a022fb-0c10-40d1-8b21-7462bbef7c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:16:19.868466 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.868422    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e685140-ec23-4d07-8a46-981e420f673f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e685140-ec23-4d07-8a46-981e420f673f" (UID: "7e685140-ec23-4d07-8a46-981e420f673f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:16:19.870011 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.869993    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a022fb-0c10-40d1-8b21-7462bbef7c62-kube-api-access-j2gm2" (OuterVolumeSpecName: "kube-api-access-j2gm2") pod "44a022fb-0c10-40d1-8b21-7462bbef7c62" (UID: "44a022fb-0c10-40d1-8b21-7462bbef7c62"). InnerVolumeSpecName "kube-api-access-j2gm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:16:19.870073 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.870018    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e685140-ec23-4d07-8a46-981e420f673f-kube-api-access-974tn" (OuterVolumeSpecName: "kube-api-access-974tn") pod "7e685140-ec23-4d07-8a46-981e420f673f" (UID: "7e685140-ec23-4d07-8a46-981e420f673f"). InnerVolumeSpecName "kube-api-access-974tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:16:19.952559 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.952470    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:19.969739 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.969698    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e685140-ec23-4d07-8a46-981e420f673f-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:19.969739 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.969731    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2gm2\" (UniqueName: \"kubernetes.io/projected/44a022fb-0c10-40d1-8b21-7462bbef7c62-kube-api-access-j2gm2\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:19.969739 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.969740    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a022fb-0c10-40d1-8b21-7462bbef7c62-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:19.969739 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:19.969749    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974tn\" (UniqueName: \"kubernetes.io/projected/7e685140-ec23-4d07-8a46-981e420f673f-kube-api-access-974tn\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:20.501687 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.501600    2586 generic.go:356] "Generic (PLEG): container finished" podID="44a022fb-0c10-40d1-8b21-7462bbef7c62" containerID="52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779" exitCode=0
Jul 29 01:16:20.501687 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.501652    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"
Jul 29 01:16:20.502230 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.501696    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk" event={"ID":"44a022fb-0c10-40d1-8b21-7462bbef7c62","Type":"ContainerDied","Data":"52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779"}
Jul 29 01:16:20.502230 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.501760    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk" event={"ID":"44a022fb-0c10-40d1-8b21-7462bbef7c62","Type":"ContainerDied","Data":"a3c725e6c695c92996caaa67659798412fb9e9775670151b7cc2e9fb0ecb3ab3"}
Jul 29 01:16:20.502230 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.501790    2586 scope.go:117] "RemoveContainer" containerID="52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779"
Jul 29 01:16:20.503175 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.503151    2586 generic.go:356] "Generic (PLEG): container finished" podID="7e685140-ec23-4d07-8a46-981e420f673f" containerID="ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4" exitCode=0
Jul 29 01:16:20.503335 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.503212    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts" event={"ID":"7e685140-ec23-4d07-8a46-981e420f673f","Type":"ContainerDied","Data":"ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4"}
Jul 29 01:16:20.503335 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.503238    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"
Jul 29 01:16:20.503335 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.503270    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts" event={"ID":"7e685140-ec23-4d07-8a46-981e420f673f","Type":"ContainerDied","Data":"ab4d5d521fffbf973a4ec17fa12ff555ad154a7a0d5c8f4dac8e84ded86c3710"}
Jul 29 01:16:20.504807 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.504780    2586 generic.go:356] "Generic (PLEG): container finished" podID="26ace133-9720-4260-a4b1-04518c8f0599" containerID="4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826" exitCode=0
Jul 29 01:16:20.504922 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.504823    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"
Jul 29 01:16:20.504922 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.504830    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4" event={"ID":"26ace133-9720-4260-a4b1-04518c8f0599","Type":"ContainerDied","Data":"4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826"}
Jul 29 01:16:20.504922 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.504877    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4" event={"ID":"26ace133-9720-4260-a4b1-04518c8f0599","Type":"ContainerDied","Data":"6cebe17c30e7ebd63f129c45be997402c3a21aecf94b79a5d521f392c01faa2a"}
Jul 29 01:16:20.512613 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.512595    2586 scope.go:117] "RemoveContainer" containerID="52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779"
Jul 29 01:16:20.512931 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:16:20.512911    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779\": container with ID starting with 52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779 not found: ID does not exist" containerID="52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779"
Jul 29 01:16:20.512976 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.512940    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779"} err="failed to get container status \"52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779\": rpc error: code = NotFound desc = could not find container \"52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779\": container with ID starting with 52c23926b9dacd88c34e4fb5a747930a6a8bd36115c042490e3d8fd10efb4779 not found: ID does not exist"
Jul 29 01:16:20.512976 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.512958    2586 scope.go:117] "RemoveContainer" containerID="ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4"
Jul 29 01:16:20.521445 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.521417    2586 scope.go:117] "RemoveContainer" containerID="ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4"
Jul 29 01:16:20.521888 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:16:20.521863    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4\": container with ID starting with ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4 not found: ID does not exist" containerID="ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4"
Jul 29 01:16:20.522010 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.521899    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4"} err="failed to get container status \"ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4\": rpc error: code = NotFound desc = could not find container \"ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4\": container with ID starting with ffcfa9c2fbe5f20ddac9bf0a9573ca80250342496bd8f946dcadd18b818ed7e4 not found: ID does not exist"
Jul 29 01:16:20.522010 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.521923    2586 scope.go:117] "RemoveContainer" containerID="4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826"
Jul 29 01:16:20.525154 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.525131    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"]
Jul 29 01:16:20.529910 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.529884    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4650f4-c66b4578-hcxts"]
Jul 29 01:16:20.531832 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.531810    2586 scope.go:117] "RemoveContainer" containerID="4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826"
Jul 29 01:16:20.532122 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:16:20.532101    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826\": container with ID starting with 4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826 not found: ID does not exist" containerID="4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826"
Jul 29 01:16:20.532176 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.532130    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826"} err="failed to get container status \"4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826\": rpc error: code = NotFound desc = could not find container \"4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826\": container with ID starting with 4a7eab3ba03f2ef2b7d99de6d75ff48959e0979c7cff457821960843f3283826 not found: ID does not exist"
Jul 29 01:16:20.538884 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.538858    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"]
Jul 29 01:16:20.542549 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.542526    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4650f4-74447cb4c5-m85h4"]
Jul 29 01:16:20.552984 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.552956    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"]
Jul 29 01:16:20.556416 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:20.556390    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4650f4-865bd8c544-g59rk"]
Jul 29 01:16:22.222129 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:22.222096    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ace133-9720-4260-a4b1-04518c8f0599" path="/var/lib/kubelet/pods/26ace133-9720-4260-a4b1-04518c8f0599/volumes"
Jul 29 01:16:22.222560 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:22.222506    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a022fb-0c10-40d1-8b21-7462bbef7c62" path="/var/lib/kubelet/pods/44a022fb-0c10-40d1-8b21-7462bbef7c62/volumes"
Jul 29 01:16:22.222837 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:22.222824    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e685140-ec23-4d07-8a46-981e420f673f" path="/var/lib/kubelet/pods/7e685140-ec23-4d07-8a46-981e420f673f/volumes"
Jul 29 01:16:25.608516 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:25.608478    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:25.712647 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:25.712615    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:25.860152 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:25.860058    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:26.092373 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.092316    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:26.210249 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210154    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"]
Jul 29 01:16:26.210601 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210587    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ace133-9720-4260-a4b1-04518c8f0599" containerName="maas-api"
Jul 29 01:16:26.210647 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210604    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ace133-9720-4260-a4b1-04518c8f0599" containerName="maas-api"
Jul 29 01:16:26.210647 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210619    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a022fb-0c10-40d1-8b21-7462bbef7c62" containerName="payload-pre-processing"
Jul 29 01:16:26.210647 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210625    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a022fb-0c10-40d1-8b21-7462bbef7c62" containerName="payload-pre-processing"
Jul 29 01:16:26.210647 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210644    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e685140-ec23-4d07-8a46-981e420f673f" containerName="payload-processing"
Jul 29 01:16:26.210771 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210651    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e685140-ec23-4d07-8a46-981e420f673f" containerName="payload-processing"
Jul 29 01:16:26.210771 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210715    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a022fb-0c10-40d1-8b21-7462bbef7c62" containerName="payload-pre-processing"
Jul 29 01:16:26.210771 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210728    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e685140-ec23-4d07-8a46-981e420f673f" containerName="payload-processing"
Jul 29 01:16:26.210771 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.210735    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ace133-9720-4260-a4b1-04518c8f0599" containerName="maas-api"
Jul 29 01:16:26.215844 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.215816    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"
Jul 29 01:16:26.218386 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.218365    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:16:26.224131 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.224104    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"]
Jul 29 01:16:26.324163 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.324122    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf8c441-1856-4c51-9c4e-1bac2c2c36d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7\" (UID: \"3bf8c441-1856-4c51-9c4e-1bac2c2c36d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"
Jul 29 01:16:26.425051 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.425008    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf8c441-1856-4c51-9c4e-1bac2c2c36d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7\" (UID: \"3bf8c441-1856-4c51-9c4e-1bac2c2c36d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"
Jul 29 01:16:26.425687 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.425664    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf8c441-1856-4c51-9c4e-1bac2c2c36d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7\" (UID: \"3bf8c441-1856-4c51-9c4e-1bac2c2c36d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"
Jul 29 01:16:26.528679 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.528586    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"
Jul 29 01:16:26.678578 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:26.678540    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"]
Jul 29 01:16:26.680266 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:16:26.680231    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf8c441_1856_4c51_9c4e_1bac2c2c36d0.slice/crio-52a119109b0d396137bdcf9eab9fb49135185967f8ceadf16800d181dfa829f8 WatchSource:0}: Error finding container 52a119109b0d396137bdcf9eab9fb49135185967f8ceadf16800d181dfa829f8: Status 404 returned error can't find the container with id 52a119109b0d396137bdcf9eab9fb49135185967f8ceadf16800d181dfa829f8
Jul 29 01:16:27.550046 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:27.550012    2586 generic.go:356] "Generic (PLEG): container finished" podID="3bf8c441-1856-4c51-9c4e-1bac2c2c36d0" containerID="36498a9e3eee0693cd4880d0005a308f5288fdc75943059847a73f9e5be22398" exitCode=0
Jul 29 01:16:27.550250 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:27.550108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7" event={"ID":"3bf8c441-1856-4c51-9c4e-1bac2c2c36d0","Type":"ContainerDied","Data":"36498a9e3eee0693cd4880d0005a308f5288fdc75943059847a73f9e5be22398"}
Jul 29 01:16:27.550250 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:27.550160    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7" event={"ID":"3bf8c441-1856-4c51-9c4e-1bac2c2c36d0","Type":"ContainerStarted","Data":"52a119109b0d396137bdcf9eab9fb49135185967f8ceadf16800d181dfa829f8"}
Jul 29 01:16:28.701580 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:28.701547    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"
Jul 29 01:16:28.846756 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:28.846722    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf8c441-1856-4c51-9c4e-1bac2c2c36d0-maas-api-service-ca\") pod \"3bf8c441-1856-4c51-9c4e-1bac2c2c36d0\" (UID: \"3bf8c441-1856-4c51-9c4e-1bac2c2c36d0\") "
Jul 29 01:16:28.847133 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:28.847107    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf8c441-1856-4c51-9c4e-1bac2c2c36d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bf8c441-1856-4c51-9c4e-1bac2c2c36d0" (UID: "3bf8c441-1856-4c51-9c4e-1bac2c2c36d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:16:28.947828 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:28.947785    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf8c441-1856-4c51-9c4e-1bac2c2c36d0-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:29.563380 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:29.563343    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7" event={"ID":"3bf8c441-1856-4c51-9c4e-1bac2c2c36d0","Type":"ContainerDied","Data":"52a119109b0d396137bdcf9eab9fb49135185967f8ceadf16800d181dfa829f8"}
Jul 29 01:16:29.563380 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:29.563371    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"
Jul 29 01:16:29.563592 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:29.563398    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a119109b0d396137bdcf9eab9fb49135185967f8ceadf16800d181dfa829f8"
Jul 29 01:16:32.790545 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:32.790460    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:41.667323 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.667280    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"]
Jul 29 01:16:41.668241 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.668144    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv" podUID="14b8312f-0e9c-47dd-8102-813f4663f1e6" containerName="maas-api" containerID="cri-o://40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448" gracePeriod=30
Jul 29 01:16:41.692224 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.692173    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:41.719104 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.719064    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"]
Jul 29 01:16:41.719397 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.719367    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59" podUID="d0a0b745-5b3a-4376-95f5-3ffc6e5b3553" containerName="payload-processing" containerID="cri-o://5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04" gracePeriod=30
Jul 29 01:16:41.724386 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.724354    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"]
Jul 29 01:16:41.724621 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.724592    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc" podUID="61ea29dd-d1ee-45df-8155-e610a153e714" containerName="payload-pre-processing" containerID="cri-o://3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549" gracePeriod=30
Jul 29 01:16:41.761306 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.761180    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:41.821922 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.821894    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:41.872909 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.872719    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"]
Jul 29 01:16:41.875958 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.875916    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d5d9a9-t5wh7"]
Jul 29 01:16:41.879350 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:41.879299    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:42.000075 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.000047    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:16:42.016248 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.016184    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:16:42.024652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.024624    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:16:42.167285 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.167243    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b8312f-0e9c-47dd-8102-813f4663f1e6-openshift-service-ca\") pod \"14b8312f-0e9c-47dd-8102-813f4663f1e6\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") "
Jul 29 01:16:42.167494 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.167308    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svnwg\" (UniqueName: \"kubernetes.io/projected/14b8312f-0e9c-47dd-8102-813f4663f1e6-kube-api-access-svnwg\") pod \"14b8312f-0e9c-47dd-8102-813f4663f1e6\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") "
Jul 29 01:16:42.167494 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.167354    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-plugins-config-volume\") pod \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\" (UID: \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\") "
Jul 29 01:16:42.167494 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.167391    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b8312f-0e9c-47dd-8102-813f4663f1e6-maas-api-tls\") pod \"14b8312f-0e9c-47dd-8102-813f4663f1e6\" (UID: \"14b8312f-0e9c-47dd-8102-813f4663f1e6\") "
Jul 29 01:16:42.167494 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.167429    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ea29dd-d1ee-45df-8155-e610a153e714-plugins-config-volume\") pod \"61ea29dd-d1ee-45df-8155-e610a153e714\" (UID: \"61ea29dd-d1ee-45df-8155-e610a153e714\") "
Jul 29 01:16:42.167494 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.167474    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69r6b\" (UniqueName: \"kubernetes.io/projected/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-kube-api-access-69r6b\") pod \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\" (UID: \"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553\") "
Jul 29 01:16:42.167770 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.167514    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jfsp\" (UniqueName: \"kubernetes.io/projected/61ea29dd-d1ee-45df-8155-e610a153e714-kube-api-access-7jfsp\") pod \"61ea29dd-d1ee-45df-8155-e610a153e714\" (UID: \"61ea29dd-d1ee-45df-8155-e610a153e714\") "
Jul 29 01:16:42.168013 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.167644    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b8312f-0e9c-47dd-8102-813f4663f1e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14b8312f-0e9c-47dd-8102-813f4663f1e6" (UID: "14b8312f-0e9c-47dd-8102-813f4663f1e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:16:42.168134 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.168031    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ea29dd-d1ee-45df-8155-e610a153e714-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61ea29dd-d1ee-45df-8155-e610a153e714" (UID: "61ea29dd-d1ee-45df-8155-e610a153e714"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:16:42.168134 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.168086    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0a0b745-5b3a-4376-95f5-3ffc6e5b3553" (UID: "d0a0b745-5b3a-4376-95f5-3ffc6e5b3553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:16:42.168302 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.168275    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b8312f-0e9c-47dd-8102-813f4663f1e6-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:42.168302 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.168303    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:42.168454 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.168317    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ea29dd-d1ee-45df-8155-e610a153e714-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:42.169682 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.169659    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b8312f-0e9c-47dd-8102-813f4663f1e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14b8312f-0e9c-47dd-8102-813f4663f1e6" (UID: "14b8312f-0e9c-47dd-8102-813f4663f1e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:16:42.169759 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.169673    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ea29dd-d1ee-45df-8155-e610a153e714-kube-api-access-7jfsp" (OuterVolumeSpecName: "kube-api-access-7jfsp") pod "61ea29dd-d1ee-45df-8155-e610a153e714" (UID: "61ea29dd-d1ee-45df-8155-e610a153e714"). InnerVolumeSpecName "kube-api-access-7jfsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:16:42.169943 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.169922    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b8312f-0e9c-47dd-8102-813f4663f1e6-kube-api-access-svnwg" (OuterVolumeSpecName: "kube-api-access-svnwg") pod "14b8312f-0e9c-47dd-8102-813f4663f1e6" (UID: "14b8312f-0e9c-47dd-8102-813f4663f1e6"). InnerVolumeSpecName "kube-api-access-svnwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:16:42.169943 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.169925    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-kube-api-access-69r6b" (OuterVolumeSpecName: "kube-api-access-69r6b") pod "d0a0b745-5b3a-4376-95f5-3ffc6e5b3553" (UID: "d0a0b745-5b3a-4376-95f5-3ffc6e5b3553"). InnerVolumeSpecName "kube-api-access-69r6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:16:42.222146 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.222065    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf8c441-1856-4c51-9c4e-1bac2c2c36d0" path="/var/lib/kubelet/pods/3bf8c441-1856-4c51-9c4e-1bac2c2c36d0/volumes"
Jul 29 01:16:42.248696 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.248667    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:42.269780 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.269738    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfsp\" (UniqueName: \"kubernetes.io/projected/61ea29dd-d1ee-45df-8155-e610a153e714-kube-api-access-7jfsp\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:42.269780 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.269773    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svnwg\" (UniqueName: \"kubernetes.io/projected/14b8312f-0e9c-47dd-8102-813f4663f1e6-kube-api-access-svnwg\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:42.269780 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.269783    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b8312f-0e9c-47dd-8102-813f4663f1e6-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:42.269780 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.269792    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69r6b\" (UniqueName: \"kubernetes.io/projected/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553-kube-api-access-69r6b\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:16:42.638069 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.638031    2586 generic.go:356] "Generic (PLEG): container finished" podID="14b8312f-0e9c-47dd-8102-813f4663f1e6" containerID="40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448" exitCode=0
Jul 29 01:16:42.638326 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.638076    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"
Jul 29 01:16:42.638326 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.638129    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv" event={"ID":"14b8312f-0e9c-47dd-8102-813f4663f1e6","Type":"ContainerDied","Data":"40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448"}
Jul 29 01:16:42.638326 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.638189    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv" event={"ID":"14b8312f-0e9c-47dd-8102-813f4663f1e6","Type":"ContainerDied","Data":"494a21628e6429460610f5ff920cdb64e14403cb8a0f10cca752236799adbf0b"}
Jul 29 01:16:42.638326 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.638227    2586 scope.go:117] "RemoveContainer" containerID="40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448"
Jul 29 01:16:42.639869 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.639843    2586 generic.go:356] "Generic (PLEG): container finished" podID="d0a0b745-5b3a-4376-95f5-3ffc6e5b3553" containerID="5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04" exitCode=0
Jul 29 01:16:42.639976 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.639881    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"
Jul 29 01:16:42.639976 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.639923    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59" event={"ID":"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553","Type":"ContainerDied","Data":"5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04"}
Jul 29 01:16:42.640080 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.639969    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59" event={"ID":"d0a0b745-5b3a-4376-95f5-3ffc6e5b3553","Type":"ContainerDied","Data":"15bc6e54fd61912ac3b5978ed67547645bce7f4011d64cc316f8bf001f0c7223"}
Jul 29 01:16:42.641906 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.641878    2586 generic.go:356] "Generic (PLEG): container finished" podID="61ea29dd-d1ee-45df-8155-e610a153e714" containerID="3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549" exitCode=0
Jul 29 01:16:42.642023 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.641937    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc" event={"ID":"61ea29dd-d1ee-45df-8155-e610a153e714","Type":"ContainerDied","Data":"3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549"}
Jul 29 01:16:42.642023 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.641970    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"
Jul 29 01:16:42.642023 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.641979    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc" event={"ID":"61ea29dd-d1ee-45df-8155-e610a153e714","Type":"ContainerDied","Data":"537e78516c54fc7a566651407c412672e6aa0ed6adf0bf2c111810178933c86b"}
Jul 29 01:16:42.649351 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.649332    2586 scope.go:117] "RemoveContainer" containerID="40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448"
Jul 29 01:16:42.649626 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:16:42.649608    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448\": container with ID starting with 40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448 not found: ID does not exist" containerID="40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448"
Jul 29 01:16:42.649718 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.649640    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448"} err="failed to get container status \"40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448\": rpc error: code = NotFound desc = could not find container \"40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448\": container with ID starting with 40904750f6ffb8ae8251770ac9bdef8bba9de987054149d3de6779211078f448 not found: ID does not exist"
Jul 29 01:16:42.649718 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.649662    2586 scope.go:117] "RemoveContainer" containerID="5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04"
Jul 29 01:16:42.660091 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.660062    2586 scope.go:117] "RemoveContainer" containerID="5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04"
Jul 29 01:16:42.660560 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:16:42.660537    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04\": container with ID starting with 5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04 not found: ID does not exist" containerID="5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04"
Jul 29 01:16:42.660673 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.660570    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04"} err="failed to get container status \"5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04\": rpc error: code = NotFound desc = could not find container \"5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04\": container with ID starting with 5414a9fe64add5baf46e735b31a5ef15f2fe2c3892d9e8cfa9716ead1a21ec04 not found: ID does not exist"
Jul 29 01:16:42.660673 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.660595    2586 scope.go:117] "RemoveContainer" containerID="3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549"
Jul 29 01:16:42.661957 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.661925    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"]
Jul 29 01:16:42.668106 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.668074    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d5d9a9-55475bd4b-74b59"]
Jul 29 01:16:42.670463 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.670304    2586 scope.go:117] "RemoveContainer" containerID="3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549"
Jul 29 01:16:42.670643 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:16:42.670623    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549\": container with ID starting with 3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549 not found: ID does not exist" containerID="3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549"
Jul 29 01:16:42.670710 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.670651    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549"} err="failed to get container status \"3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549\": rpc error: code = NotFound desc = could not find container \"3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549\": container with ID starting with 3e527b59461ba39467194a83533c08b61153f9c1e81565001cef36048909c549 not found: ID does not exist"
Jul 29 01:16:42.677124 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.677098    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"]
Jul 29 01:16:42.680998 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.680972    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d5d9a9-589b7bc547-8dvwv"]
Jul 29 01:16:42.692720 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.692694    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"]
Jul 29 01:16:42.697533 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:42.697505    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d5d9a9-7c7fccf984-v22cc"]
Jul 29 01:16:44.221852 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:44.221816    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14b8312f-0e9c-47dd-8102-813f4663f1e6" path="/var/lib/kubelet/pods/14b8312f-0e9c-47dd-8102-813f4663f1e6/volumes"
Jul 29 01:16:44.222269 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:44.222179    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ea29dd-d1ee-45df-8155-e610a153e714" path="/var/lib/kubelet/pods/61ea29dd-d1ee-45df-8155-e610a153e714/volumes"
Jul 29 01:16:44.222536 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:44.222522    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a0b745-5b3a-4376-95f5-3ffc6e5b3553" path="/var/lib/kubelet/pods/d0a0b745-5b3a-4376-95f5-3ffc6e5b3553/volumes"
Jul 29 01:16:48.025889 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:48.025854    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:48.087797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:48.087753    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:48.300037 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:48.299936    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:48.518031 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:48.517994    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:48.861405 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:48.861366    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:49.162676 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:49.162574    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:49.584247 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:49.584187    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:52.477557 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.477509    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:52.488161 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488123    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"]
Jul 29 01:16:52.488776 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488744    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b8312f-0e9c-47dd-8102-813f4663f1e6" containerName="maas-api"
Jul 29 01:16:52.488776 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488772    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b8312f-0e9c-47dd-8102-813f4663f1e6" containerName="maas-api"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488789    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a0b745-5b3a-4376-95f5-3ffc6e5b3553" containerName="payload-processing"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488800    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a0b745-5b3a-4376-95f5-3ffc6e5b3553" containerName="payload-processing"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488822    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf8c441-1856-4c51-9c4e-1bac2c2c36d0" containerName="revoke-keys"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488832    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf8c441-1856-4c51-9c4e-1bac2c2c36d0" containerName="revoke-keys"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488846    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ea29dd-d1ee-45df-8155-e610a153e714" containerName="payload-pre-processing"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488854    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ea29dd-d1ee-45df-8155-e610a153e714" containerName="payload-pre-processing"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488938    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ea29dd-d1ee-45df-8155-e610a153e714" containerName="payload-pre-processing"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488948    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a0b745-5b3a-4376-95f5-3ffc6e5b3553" containerName="payload-processing"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488960    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="14b8312f-0e9c-47dd-8102-813f4663f1e6" containerName="maas-api"
Jul 29 01:16:52.488977 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.488976    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf8c441-1856-4c51-9c4e-1bac2c2c36d0" containerName="revoke-keys"
Jul 29 01:16:52.492058 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.492039    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.495295 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.495244    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 01:16:52.495455 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.495323    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ebcf3a-serving-cert\""
Jul 29 01:16:52.495455 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.495327    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:16:52.496538 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.496518    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9jxwk\""
Jul 29 01:16:52.505924 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.505897    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"]
Jul 29 01:16:52.556893 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.556852    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65fz4\" (UniqueName: \"kubernetes.io/projected/219b1bdb-1798-4a4f-9718-8c15291fc742-kube-api-access-65fz4\") pod \"maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.557087 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.556913    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219b1bdb-1798-4a4f-9718-8c15291fc742-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.557087 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.557050    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219b1bdb-1798-4a4f-9718-8c15291fc742-maas-api-tls\") pod \"maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.559222 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.559171    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:52.658541 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.658505    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65fz4\" (UniqueName: \"kubernetes.io/projected/219b1bdb-1798-4a4f-9718-8c15291fc742-kube-api-access-65fz4\") pod \"maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.658742 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.658568    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219b1bdb-1798-4a4f-9718-8c15291fc742-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.658742 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.658629    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219b1bdb-1798-4a4f-9718-8c15291fc742-maas-api-tls\") pod \"maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.659321 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.659295    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219b1bdb-1798-4a4f-9718-8c15291fc742-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.661223 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.661181    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219b1bdb-1798-4a4f-9718-8c15291fc742-maas-api-tls\") pod \"maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.666902 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.666874    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:52.676856 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.676825    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65fz4\" (UniqueName: \"kubernetes.io/projected/219b1bdb-1798-4a4f-9718-8c15291fc742-kube-api-access-65fz4\") pod \"maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.804078 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.803991    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:52.964701 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:52.964678    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"]
Jul 29 01:16:52.966911 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:16:52.966875    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219b1bdb_1798_4a4f_9718_8c15291fc742.slice/crio-70dc7e56732735ced647b7cd6f0fde2100a29619c4b0aececc8aa839f64a2b35 WatchSource:0}: Error finding container 70dc7e56732735ced647b7cd6f0fde2100a29619c4b0aececc8aa839f64a2b35: Status 404 returned error can't find the container with id 70dc7e56732735ced647b7cd6f0fde2100a29619c4b0aececc8aa839f64a2b35
Jul 29 01:16:53.084972 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.084940    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:53.172014 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.171971    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:53.172965 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.172940    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"]
Jul 29 01:16:53.177735 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.177713    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:16:53.180474 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.180454    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-ebcf3a\""
Jul 29 01:16:53.180606 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.180586    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-ebcf3a-dockercfg-pqvbk\""
Jul 29 01:16:53.186901 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.186755    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"]
Jul 29 01:16:53.216727 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.216691    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"]
Jul 29 01:16:53.219948 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.219932    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:16:53.231332 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.231307    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"]
Jul 29 01:16:53.264680 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.264643    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a7bad0-3e01-469f-85af-3b104ab576a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l\" (UID: \"b6a7bad0-3e01-469f-85af-3b104ab576a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:16:53.264870 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.264747    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7m6\" (UniqueName: \"kubernetes.io/projected/b6a7bad0-3e01-469f-85af-3b104ab576a1-kube-api-access-gt7m6\") pod \"payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l\" (UID: \"b6a7bad0-3e01-469f-85af-3b104ab576a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:16:53.264870 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.264781    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7hx\" (UniqueName: \"kubernetes.io/projected/230230a8-349a-4d99-94f9-428fef25f0a2-kube-api-access-8z7hx\") pod \"payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9\" (UID: \"230230a8-349a-4d99-94f9-428fef25f0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:16:53.265083 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.265062    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230230a8-349a-4d99-94f9-428fef25f0a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9\" (UID: \"230230a8-349a-4d99-94f9-428fef25f0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:16:53.366326 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.366190    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230230a8-349a-4d99-94f9-428fef25f0a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9\" (UID: \"230230a8-349a-4d99-94f9-428fef25f0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:16:53.366326 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.366297    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a7bad0-3e01-469f-85af-3b104ab576a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l\" (UID: \"b6a7bad0-3e01-469f-85af-3b104ab576a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:16:53.366571 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.366336    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7m6\" (UniqueName: \"kubernetes.io/projected/b6a7bad0-3e01-469f-85af-3b104ab576a1-kube-api-access-gt7m6\") pod \"payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l\" (UID: \"b6a7bad0-3e01-469f-85af-3b104ab576a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:16:53.366571 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.366359    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7hx\" (UniqueName: \"kubernetes.io/projected/230230a8-349a-4d99-94f9-428fef25f0a2-kube-api-access-8z7hx\") pod \"payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9\" (UID: \"230230a8-349a-4d99-94f9-428fef25f0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:16:53.367033 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.367006    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230230a8-349a-4d99-94f9-428fef25f0a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9\" (UID: \"230230a8-349a-4d99-94f9-428fef25f0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:16:53.367419 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.367394    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a7bad0-3e01-469f-85af-3b104ab576a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l\" (UID: \"b6a7bad0-3e01-469f-85af-3b104ab576a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:16:53.374613 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.374584    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7m6\" (UniqueName: \"kubernetes.io/projected/b6a7bad0-3e01-469f-85af-3b104ab576a1-kube-api-access-gt7m6\") pod \"payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l\" (UID: \"b6a7bad0-3e01-469f-85af-3b104ab576a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:16:53.374727 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.374710    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7hx\" (UniqueName: \"kubernetes.io/projected/230230a8-349a-4d99-94f9-428fef25f0a2-kube-api-access-8z7hx\") pod \"payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9\" (UID: \"230230a8-349a-4d99-94f9-428fef25f0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:16:53.491536 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.491493    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:16:53.531661 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.531627    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:16:53.658463 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.657069    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"]
Jul 29 01:16:53.714138 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.714102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v" event={"ID":"219b1bdb-1798-4a4f-9718-8c15291fc742","Type":"ContainerStarted","Data":"811251b429b22c5f281870ad1c1ee6a5e6886f578531befbf4dd39bcd1c9a8fc"}
Jul 29 01:16:53.714308 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.714171    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:16:53.714308 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.714211    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v" event={"ID":"219b1bdb-1798-4a4f-9718-8c15291fc742","Type":"ContainerStarted","Data":"70dc7e56732735ced647b7cd6f0fde2100a29619c4b0aececc8aa839f64a2b35"}
Jul 29 01:16:53.715583 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.715551    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9" event={"ID":"230230a8-349a-4d99-94f9-428fef25f0a2","Type":"ContainerStarted","Data":"c2922b051eb1645930ee6e536df212eb01de0002f8122b2f51a83614dea59244"}
Jul 29 01:16:53.717606 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:16:53.717577    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a7bad0_3e01_469f_85af_3b104ab576a1.slice/crio-32d70f6b1be13f5d28c29663d88623672b26dca39665ce7074c6cf7fa521be73 WatchSource:0}: Error finding container 32d70f6b1be13f5d28c29663d88623672b26dca39665ce7074c6cf7fa521be73: Status 404 returned error can't find the container with id 32d70f6b1be13f5d28c29663d88623672b26dca39665ce7074c6cf7fa521be73
Jul 29 01:16:53.717716 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.717663    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"]
Jul 29 01:16:53.732589 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:53.732540    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v" podStartSLOduration=1.166531425 podStartE2EDuration="1.732522897s" totalImagesPullingTime="565.991472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:16:53.730062791 +0000 UTC m=+2444.038990632" watchObservedRunningTime="2026-07-29 01:16:53.732522897 +0000 UTC m=+2444.041450773"
Jul 29 01:16:54.723554 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:54.723516    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l" event={"ID":"b6a7bad0-3e01-469f-85af-3b104ab576a1","Type":"ContainerStarted","Data":"367c530613afc2c27a4b0371eaf5f79aa1b6c8ef4194d715f6c31b326f5d5379"}
Jul 29 01:16:54.724024 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:54.723569    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l" event={"ID":"b6a7bad0-3e01-469f-85af-3b104ab576a1","Type":"ContainerStarted","Data":"32d70f6b1be13f5d28c29663d88623672b26dca39665ce7074c6cf7fa521be73"}
Jul 29 01:16:54.724024 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:54.723624    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:16:54.731785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:54.731748    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9" event={"ID":"230230a8-349a-4d99-94f9-428fef25f0a2","Type":"ContainerStarted","Data":"12db715f021fdaf62ed967e27ad7114452b71d9ca25fed7543b6e7700799a20b"}
Jul 29 01:16:54.731955 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:54.731940    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:16:54.746915 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:54.746868    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l" podStartSLOduration=1.746853719 podStartE2EDuration="1.746853719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:16:54.744335209 +0000 UTC m=+2445.053263033" watchObservedRunningTime="2026-07-29 01:16:54.746853719 +0000 UTC m=+2445.055781561"
Jul 29 01:16:54.758867 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:54.758815    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9" podStartSLOduration=1.758797386 podStartE2EDuration="1.758797386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:16:54.758035009 +0000 UTC m=+2445.066962852" watchObservedRunningTime="2026-07-29 01:16:54.758797386 +0000 UTC m=+2445.067725229"
Jul 29 01:16:58.947107 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:58.947073    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:16:59.737206 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:16:59.737163    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:17:05.740411 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:05.740378    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:17:05.740814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:05.740455    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:17:17.334429 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:17.334393    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:17.657035 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:17.656890    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:17.958302 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:17.958219    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:18.519501 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:18.519462    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:20.503785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.503689    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:20.607099 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.607061    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:20.608891 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.608863    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"]
Jul 29 01:17:20.612620 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.612600    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:20.614939 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.614916    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-afd6d5-serving-cert\""
Jul 29 01:17:20.625478 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.625435    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"]
Jul 29 01:17:20.703423 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.703388    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:20.715506 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.715470    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-maas-api-tls\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:20.715506 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.715523    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:20.715506 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.715550    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mjg\" (UniqueName: \"kubernetes.io/projected/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-kube-api-access-m4mjg\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:20.816751 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.816706    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-maas-api-tls\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:20.816961 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.816783    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:20.816961 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.816816    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mjg\" (UniqueName: \"kubernetes.io/projected/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-kube-api-access-m4mjg\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:20.816961 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:17:20.816884    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-serving-cert: secret "maas-api-e2e-shared-b-afd6d5-serving-cert" not found
Jul 29 01:17:20.817134 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:17:20.816966    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-maas-api-tls podName:25cc489d-dcc6-42f9-8a75-02d55af2bdfc nodeName:}" failed. No retries permitted until 2026-07-29 01:17:21.316948398 +0000 UTC m=+2471.625876218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-maas-api-tls") pod "maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf" (UID: "25cc489d-dcc6-42f9-8a75-02d55af2bdfc") : secret "maas-api-e2e-shared-b-afd6d5-serving-cert" not found
Jul 29 01:17:20.817453 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.817434    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:20.825620 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.825593    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mjg\" (UniqueName: \"kubernetes.io/projected/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-kube-api-access-m4mjg\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:20.957972 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:20.957937    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:21.150360 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.150264    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:21.245319 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.245285    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:21.246981 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.246959    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"]
Jul 29 01:17:21.250626 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.250610    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:21.255791 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.255766    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-afd6d5-dockercfg-l4xv4\""
Jul 29 01:17:21.255917 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.255848    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-afd6d5\""
Jul 29 01:17:21.260712 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.260689    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"]
Jul 29 01:17:21.289089 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.289057    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"]
Jul 29 01:17:21.292673 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.292656    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:21.316495 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.316461    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"]
Jul 29 01:17:21.321284 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.321256    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-maas-api-tls\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:21.321429 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.321289    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5x95\" (UniqueName: \"kubernetes.io/projected/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-kube-api-access-f5x95\") pod \"payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8\" (UID: \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:21.321429 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.321364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8\" (UID: \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:21.323602 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.323578    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-maas-api-tls\") pod \"maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:21.422453 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.422345    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5x95\" (UniqueName: \"kubernetes.io/projected/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-kube-api-access-f5x95\") pod \"payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8\" (UID: \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:21.422453 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.422393    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2\" (UID: \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:21.422453 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.422419    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcrd4\" (UniqueName: \"kubernetes.io/projected/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-kube-api-access-rcrd4\") pod \"payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2\" (UID: \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:21.422697 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.422506    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8\" (UID: \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:21.423143 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.423123    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8\" (UID: \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:21.436550 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.436523    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5x95\" (UniqueName: \"kubernetes.io/projected/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-kube-api-access-f5x95\") pod \"payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8\" (UID: \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:21.523210 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.523169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2\" (UID: \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:21.523683 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.523233    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcrd4\" (UniqueName: \"kubernetes.io/projected/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-kube-api-access-rcrd4\") pod \"payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2\" (UID: \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:21.523683 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.523322    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:21.523888 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.523860    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2\" (UID: \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:21.531439 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.531407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcrd4\" (UniqueName: \"kubernetes.io/projected/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-kube-api-access-rcrd4\") pod \"payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2\" (UID: \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:21.564109 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.564072    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:21.603607 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.603575    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:21.696315 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.696277    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"]
Jul 29 01:17:21.700072 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:17:21.700037    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cc489d_dcc6_42f9_8a75_02d55af2bdfc.slice/crio-55d35f8e4b59708f5957ae1856de0845d5e1faafb51bdb8fea5a10e8fea6651d WatchSource:0}: Error finding container 55d35f8e4b59708f5957ae1856de0845d5e1faafb51bdb8fea5a10e8fea6651d: Status 404 returned error can't find the container with id 55d35f8e4b59708f5957ae1856de0845d5e1faafb51bdb8fea5a10e8fea6651d
Jul 29 01:17:21.701910 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.701883    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:17:21.747591 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.747559    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"]
Jul 29 01:17:21.749260 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:17:21.749228    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29163a3c_d1ff_4f8f_b311_9ec5b0e1bd44.slice/crio-2563d27c7aed1534c5f554fff31d8328c7b7cba22066cc7cdca44a169aa8119c WatchSource:0}: Error finding container 2563d27c7aed1534c5f554fff31d8328c7b7cba22066cc7cdca44a169aa8119c: Status 404 returned error can't find the container with id 2563d27c7aed1534c5f554fff31d8328c7b7cba22066cc7cdca44a169aa8119c
Jul 29 01:17:21.789597 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.789570    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"]
Jul 29 01:17:21.792516 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:17:21.792475    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddede6f03_bcc4_4d08_8ca1_eaa2b21aa4bf.slice/crio-964d0ad12eda7dee77a206a740139edc600d11819a6ec7e6bc0d9d9992d4ea40 WatchSource:0}: Error finding container 964d0ad12eda7dee77a206a740139edc600d11819a6ec7e6bc0d9d9992d4ea40: Status 404 returned error can't find the container with id 964d0ad12eda7dee77a206a740139edc600d11819a6ec7e6bc0d9d9992d4ea40
Jul 29 01:17:21.891303 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.891253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2" event={"ID":"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf","Type":"ContainerStarted","Data":"04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11"}
Jul 29 01:17:21.891467 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.891317    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2" event={"ID":"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf","Type":"ContainerStarted","Data":"964d0ad12eda7dee77a206a740139edc600d11819a6ec7e6bc0d9d9992d4ea40"}
Jul 29 01:17:21.891467 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.891459    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:21.892688 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.892658    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf" event={"ID":"25cc489d-dcc6-42f9-8a75-02d55af2bdfc","Type":"ContainerStarted","Data":"55d35f8e4b59708f5957ae1856de0845d5e1faafb51bdb8fea5a10e8fea6651d"}
Jul 29 01:17:21.894507 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.894484    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8" event={"ID":"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44","Type":"ContainerStarted","Data":"472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7"}
Jul 29 01:17:21.894616 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.894528    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8" event={"ID":"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44","Type":"ContainerStarted","Data":"2563d27c7aed1534c5f554fff31d8328c7b7cba22066cc7cdca44a169aa8119c"}
Jul 29 01:17:21.894616 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.894559    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:21.906449 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.906401    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2" podStartSLOduration=0.906386284 podStartE2EDuration="906.386284ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:17:21.905371748 +0000 UTC m=+2472.214299590" watchObservedRunningTime="2026-07-29 01:17:21.906386284 +0000 UTC m=+2472.215314130"
Jul 29 01:17:21.923073 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:21.923023    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8" podStartSLOduration=0.923008642 podStartE2EDuration="923.008642ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:17:21.921586375 +0000 UTC m=+2472.230514219" watchObservedRunningTime="2026-07-29 01:17:21.923008642 +0000 UTC m=+2472.231936487"
Jul 29 01:17:22.901869 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:22.901831    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf" event={"ID":"25cc489d-dcc6-42f9-8a75-02d55af2bdfc","Type":"ContainerStarted","Data":"7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329"}
Jul 29 01:17:22.902407 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:22.902310    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:22.920746 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:22.920696    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf" podStartSLOduration=2.5109174640000003 podStartE2EDuration="2.920680812s" totalImagesPullingTime="409.763348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:17:22.918030967 +0000 UTC m=+2473.226958800" watchObservedRunningTime="2026-07-29 01:17:22.920680812 +0000 UTC m=+2473.229608653"
Jul 29 01:17:29.918881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:29.918842    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:17:32.903826 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:32.903793    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:17:32.904261 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:32.903866    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:17:45.511618 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:45.511575    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:17:51.343170 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:17:51.343136    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:18:23.717212 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.717171    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"]
Jul 29 01:18:23.721560 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.721537    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.725459 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.725435    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-afd6d5\"/\"openshift-service-ca.crt\""
Jul 29 01:18:23.725600 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.725552    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-afd6d5\"/\"default-dockercfg-n2r8w\""
Jul 29 01:18:23.725987 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.725935    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-afd6d5\"/\"kube-root-ca.crt\""
Jul 29 01:18:23.726177 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.726154    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-afd6d5\"/\"auth-test-model-afd6d5-kserve-self-signed-certs\""
Jul 29 01:18:23.729542 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.729518    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"]
Jul 29 01:18:23.765346 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.765321    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:18:23.811481 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.811443    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-model-cache\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.811481 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.811478    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-tmp-dir\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.811685 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.811510    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4w8w\" (UniqueName: \"kubernetes.io/projected/0d122b94-d540-4322-bfa1-67747b1aa294-kube-api-access-s4w8w\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.811685 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.811557    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d122b94-d540-4322-bfa1-67747b1aa294-tls-certs\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.811685 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.811578    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-home\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.811685 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.811595    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-dshm\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.912823 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.912788    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4w8w\" (UniqueName: \"kubernetes.io/projected/0d122b94-d540-4322-bfa1-67747b1aa294-kube-api-access-s4w8w\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.913117 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.913098    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d122b94-d540-4322-bfa1-67747b1aa294-tls-certs\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.913279 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.913263    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-home\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.913394 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.913382    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-dshm\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.913559 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.913544    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-model-cache\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.913659 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.913640    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-tmp-dir\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.914057 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.914026    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-model-cache\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.914176 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.914055    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-tmp-dir\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.914176 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.914118    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-home\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.916048 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.916026    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d122b94-d540-4322-bfa1-67747b1aa294-tls-certs\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.916147 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.916081    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-dshm\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:23.921817 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:23.921795    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4w8w\" (UniqueName: \"kubernetes.io/projected/0d122b94-d540-4322-bfa1-67747b1aa294-kube-api-access-s4w8w\") pod \"auth-test-model-afd6d5-kserve-5769bd89c9-dc47n\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") " pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:24.033243 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:24.033126    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:24.170903 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:24.170875    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"]
Jul 29 01:18:24.172452 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:18:24.172423    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d122b94_d540_4322_bfa1_67747b1aa294.slice/crio-9a5201e21f7adc482e4e240b95ec6ee054a320b4000f14689facddff01a6d55d WatchSource:0}: Error finding container 9a5201e21f7adc482e4e240b95ec6ee054a320b4000f14689facddff01a6d55d: Status 404 returned error can't find the container with id 9a5201e21f7adc482e4e240b95ec6ee054a320b4000f14689facddff01a6d55d
Jul 29 01:18:24.254576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:24.254533    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n" event={"ID":"0d122b94-d540-4322-bfa1-67747b1aa294","Type":"ContainerStarted","Data":"76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b"}
Jul 29 01:18:24.254576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:24.254577    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n" event={"ID":"0d122b94-d540-4322-bfa1-67747b1aa294","Type":"ContainerStarted","Data":"9a5201e21f7adc482e4e240b95ec6ee054a320b4000f14689facddff01a6d55d"}
Jul 29 01:18:24.272159 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:24.272109    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n" podStartSLOduration=1.272092084 podStartE2EDuration="1.272092084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:18:24.271447463 +0000 UTC m=+2534.580375307" watchObservedRunningTime="2026-07-29 01:18:24.272092084 +0000 UTC m=+2534.581019925"
Jul 29 01:18:30.069534 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:30.069473    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:18:30.171102 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:30.171064    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:18:34.033573 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:34.033530    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:34.034025 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:34.033588    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:34.046178 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:34.046148    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:34.325370 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:34.325340    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:18:44.028569 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:44.028532    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:18:45.624693 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:18:45.624653    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:20:03.731146 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:20:03.731101    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:21:12.353426 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:21:12.353394    2586 scope.go:117] "RemoveContainer" containerID="f11af363536fa578ee9b9be6a33777c9e258d4e0b9e670326ffb929c9249e23d"
Jul 29 01:21:12.362547 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:21:12.362524    2586 scope.go:117] "RemoveContainer" containerID="07ef990ef52dbb94cab87052dd5565ea0c16e764ccf8cd4b8ee1f888e088c271"
Jul 29 01:22:39.163124 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:39.163036    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:22:40.015825 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:40.015777    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"]
Jul 29 01:22:40.030308 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:40.030190    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:22:52.717363 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:52.717326    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:22:55.036163 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.036112    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n" podUID="0d122b94-d540-4322-bfa1-67747b1aa294" containerName="main" containerID="cri-o://76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b" gracePeriod=45
Jul 29 01:22:55.300834 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.300760    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:22:55.361223 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361170    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-model-cache\") pod \"0d122b94-d540-4322-bfa1-67747b1aa294\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") "
Jul 29 01:22:55.361223 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361226    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-tmp-dir\") pod \"0d122b94-d540-4322-bfa1-67747b1aa294\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") "
Jul 29 01:22:55.361501 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361251    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4w8w\" (UniqueName: \"kubernetes.io/projected/0d122b94-d540-4322-bfa1-67747b1aa294-kube-api-access-s4w8w\") pod \"0d122b94-d540-4322-bfa1-67747b1aa294\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") "
Jul 29 01:22:55.361501 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361383    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-home\") pod \"0d122b94-d540-4322-bfa1-67747b1aa294\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") "
Jul 29 01:22:55.361501 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361418    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d122b94-d540-4322-bfa1-67747b1aa294-tls-certs\") pod \"0d122b94-d540-4322-bfa1-67747b1aa294\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") "
Jul 29 01:22:55.361501 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361460    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-dshm\") pod \"0d122b94-d540-4322-bfa1-67747b1aa294\" (UID: \"0d122b94-d540-4322-bfa1-67747b1aa294\") "
Jul 29 01:22:55.361501 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361485    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d122b94-d540-4322-bfa1-67747b1aa294" (UID: "0d122b94-d540-4322-bfa1-67747b1aa294"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:22:55.361779 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361555    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d122b94-d540-4322-bfa1-67747b1aa294" (UID: "0d122b94-d540-4322-bfa1-67747b1aa294"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:22:55.361779 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361683    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-home" (OuterVolumeSpecName: "home") pod "0d122b94-d540-4322-bfa1-67747b1aa294" (UID: "0d122b94-d540-4322-bfa1-67747b1aa294"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:22:55.361898 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361828    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-model-cache\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:22:55.361898 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361844    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-tmp-dir\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:22:55.361898 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.361854    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-home\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:22:55.363802 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.363769    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d122b94-d540-4322-bfa1-67747b1aa294-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d122b94-d540-4322-bfa1-67747b1aa294" (UID: "0d122b94-d540-4322-bfa1-67747b1aa294"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:22:55.363927 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.363801    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-dshm" (OuterVolumeSpecName: "dshm") pod "0d122b94-d540-4322-bfa1-67747b1aa294" (UID: "0d122b94-d540-4322-bfa1-67747b1aa294"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:22:55.363927 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.363895    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d122b94-d540-4322-bfa1-67747b1aa294-kube-api-access-s4w8w" (OuterVolumeSpecName: "kube-api-access-s4w8w") pod "0d122b94-d540-4322-bfa1-67747b1aa294" (UID: "0d122b94-d540-4322-bfa1-67747b1aa294"). InnerVolumeSpecName "kube-api-access-s4w8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:22:55.463142 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.463101    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d122b94-d540-4322-bfa1-67747b1aa294-tls-certs\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:22:55.463142 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.463136    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d122b94-d540-4322-bfa1-67747b1aa294-dshm\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:22:55.463142 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.463145    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4w8w\" (UniqueName: \"kubernetes.io/projected/0d122b94-d540-4322-bfa1-67747b1aa294-kube-api-access-s4w8w\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:22:55.860939 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.860901    2586 generic.go:356] "Generic (PLEG): container finished" podID="0d122b94-d540-4322-bfa1-67747b1aa294" containerID="76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b" exitCode=0
Jul 29 01:22:55.861161 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.860964    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"
Jul 29 01:22:55.861161 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.860989    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n" event={"ID":"0d122b94-d540-4322-bfa1-67747b1aa294","Type":"ContainerDied","Data":"76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b"}
Jul 29 01:22:55.861161 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.861041    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n" event={"ID":"0d122b94-d540-4322-bfa1-67747b1aa294","Type":"ContainerDied","Data":"9a5201e21f7adc482e4e240b95ec6ee054a320b4000f14689facddff01a6d55d"}
Jul 29 01:22:55.861161 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.861066    2586 scope.go:117] "RemoveContainer" containerID="76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b"
Jul 29 01:22:55.871249 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.871230    2586 scope.go:117] "RemoveContainer" containerID="76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b"
Jul 29 01:22:55.871560 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:22:55.871540    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b\": container with ID starting with 76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b not found: ID does not exist" containerID="76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b"
Jul 29 01:22:55.871600 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.871572    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b"} err="failed to get container status \"76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b\": rpc error: code = NotFound desc = could not find container \"76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b\": container with ID starting with 76e31d2fbfe12e475bd379b9f64df241b0a9cfe6b3953b28ac474d4e0f95003b not found: ID does not exist"
Jul 29 01:22:55.883610 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.883577    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"]
Jul 29 01:22:55.890001 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:55.889976    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-afd6d5/auth-test-model-afd6d5-kserve-5769bd89c9-dc47n"]
Jul 29 01:22:56.222367 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:22:56.222283    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d122b94-d540-4322-bfa1-67747b1aa294" path="/var/lib/kubelet/pods/0d122b94-d540-4322-bfa1-67747b1aa294/volumes"
Jul 29 01:23:12.395916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:23:12.395880    2586 scope.go:117] "RemoveContainer" containerID="36498a9e3eee0693cd4880d0005a308f5288fdc75943059847a73f9e5be22398"
Jul 29 01:23:13.048382 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:23:13.048326    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:24:15.676728 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:15.676691    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:24:16.901339 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:16.901274    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:24:24.428264 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.428226    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"]
Jul 29 01:24:24.428755 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.428714    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d122b94-d540-4322-bfa1-67747b1aa294" containerName="main"
Jul 29 01:24:24.428755 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.428728    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d122b94-d540-4322-bfa1-67747b1aa294" containerName="main"
Jul 29 01:24:24.428834 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.428799    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d122b94-d540-4322-bfa1-67747b1aa294" containerName="main"
Jul 29 01:24:24.434442 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.433246    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.440299 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.439686    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ebcf3a\"/\"rate-test-model-ebcf3a-kserve-self-signed-certs\""
Jul 29 01:24:24.440299 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.439692    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ebcf3a\"/\"openshift-service-ca.crt\""
Jul 29 01:24:24.440299 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.439776    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ebcf3a\"/\"kube-root-ca.crt\""
Jul 29 01:24:24.440589 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.440540    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ebcf3a\"/\"default-dockercfg-ghpd6\""
Jul 29 01:24:24.443897 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.443695    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"]
Jul 29 01:24:24.484441 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.484400    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tmp-dir\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.484662 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.484465    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqqlf\" (UniqueName: \"kubernetes.io/projected/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-kube-api-access-fqqlf\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.484662 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.484487    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-dshm\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.484662 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.484567    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tls-certs\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.484662 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.484601    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-model-cache\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.484662 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.484634    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-home\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.528372 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.528336    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:24:24.585779 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.585737    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqqlf\" (UniqueName: \"kubernetes.io/projected/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-kube-api-access-fqqlf\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.585779 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.585779    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-dshm\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.586034 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.585850    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tls-certs\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.586034 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.585880    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-model-cache\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.586034 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.585926    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-home\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.586034 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.585970    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tmp-dir\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.586421 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.586388    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-model-cache\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.586421 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.586402    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-home\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.586590 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.586460    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tmp-dir\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.588816 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.588791    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-dshm\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.589066 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.589047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tls-certs\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.597697 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.597670    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqqlf\" (UniqueName: \"kubernetes.io/projected/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-kube-api-access-fqqlf\") pod \"rate-test-model-ebcf3a-kserve-6696dd5475-flt5m\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") " pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.747059 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.746972    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:24.894253 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:24.894219    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"]
Jul 29 01:24:24.895416 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:24:24.895388    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadd2eb0f_c9c7_4f2e_bffa_c0dbbe31c6d6.slice/crio-1b194812b08290b014158bf782b13863ceb77fe332b468ae9f172ca083f8b959 WatchSource:0}: Error finding container 1b194812b08290b014158bf782b13863ceb77fe332b468ae9f172ca083f8b959: Status 404 returned error can't find the container with id 1b194812b08290b014158bf782b13863ceb77fe332b468ae9f172ca083f8b959
Jul 29 01:24:25.397867 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:25.397820    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m" event={"ID":"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6","Type":"ContainerStarted","Data":"632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045"}
Jul 29 01:24:25.398046 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:25.397876    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m" event={"ID":"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6","Type":"ContainerStarted","Data":"1b194812b08290b014158bf782b13863ceb77fe332b468ae9f172ca083f8b959"}
Jul 29 01:24:25.417358 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:25.417300    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m" podStartSLOduration=1.4172838429999999 podStartE2EDuration="1.417283843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:24:25.41601931 +0000 UTC m=+2895.724947164" watchObservedRunningTime="2026-07-29 01:24:25.417283843 +0000 UTC m=+2895.726211684"
Jul 29 01:24:34.747866 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:34.747821    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:34.748512 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:34.747977    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:34.763575 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:34.763540    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:35.467893 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:35.467859    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:24:41.931865 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:24:41.931831    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:25:21.002082 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:25:21.002048    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:25:21.099694 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:25:21.099656    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:25:48.413052 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:25:48.412969    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"]
Jul 29 01:25:48.433697 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:25:48.433662    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:25:49.229498 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:25:49.229464    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:03.432711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.432649    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m" podUID="add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" containerName="main" containerID="cri-o://632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045" gracePeriod=45
Jul 29 01:26:03.696471 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.696411    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:26:03.783591 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.783550    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tmp-dir\") pod \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") "
Jul 29 01:26:03.783785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.783600    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-dshm\") pod \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") "
Jul 29 01:26:03.783854 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.783777    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tls-certs\") pod \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") "
Jul 29 01:26:03.783854 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.783825    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-home\") pod \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") "
Jul 29 01:26:03.783970 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.783860    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-model-cache\") pod \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") "
Jul 29 01:26:03.783970 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.783886    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqqlf\" (UniqueName: \"kubernetes.io/projected/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-kube-api-access-fqqlf\") pod \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\" (UID: \"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6\") "
Jul 29 01:26:03.783970 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.783854    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" (UID: "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:26:03.784102 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.784018    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-home" (OuterVolumeSpecName: "home") pod "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" (UID: "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:26:03.784149 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.784103    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" (UID: "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:26:03.784311 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.784284    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-home\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:26:03.784311 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.784303    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-model-cache\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:26:03.784311 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.784313    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tmp-dir\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:26:03.785955 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.785927    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-dshm" (OuterVolumeSpecName: "dshm") pod "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" (UID: "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:26:03.786047 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.785991    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" (UID: "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:26:03.786047 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.785993    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-kube-api-access-fqqlf" (OuterVolumeSpecName: "kube-api-access-fqqlf") pod "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" (UID: "add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6"). InnerVolumeSpecName "kube-api-access-fqqlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:26:03.885726 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.885688    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqqlf\" (UniqueName: \"kubernetes.io/projected/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-kube-api-access-fqqlf\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:26:03.885726 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.885720    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-dshm\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:26:03.885726 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.885730    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6-tls-certs\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:26:03.977512 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.977431    2586 generic.go:356] "Generic (PLEG): container finished" podID="add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" containerID="632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045" exitCode=0
Jul 29 01:26:03.977512 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.977472    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m" event={"ID":"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6","Type":"ContainerDied","Data":"632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045"}
Jul 29 01:26:03.977512 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.977500    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m" event={"ID":"add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6","Type":"ContainerDied","Data":"1b194812b08290b014158bf782b13863ceb77fe332b468ae9f172ca083f8b959"}
Jul 29 01:26:03.977785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.977516    2586 scope.go:117] "RemoveContainer" containerID="632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045"
Jul 29 01:26:03.977785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.977472    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"
Jul 29 01:26:03.987483 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.987461    2586 scope.go:117] "RemoveContainer" containerID="632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045"
Jul 29 01:26:03.987771 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:26:03.987750    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045\": container with ID starting with 632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045 not found: ID does not exist" containerID="632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045"
Jul 29 01:26:03.987821 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:03.987781    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045"} err="failed to get container status \"632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045\": rpc error: code = NotFound desc = could not find container \"632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045\": container with ID starting with 632aecca2d2057b1379cefdc738c0c12cd6a75452bac1d7e0b8a6e048f437045 not found: ID does not exist"
Jul 29 01:26:04.000159 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:04.000128    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"]
Jul 29 01:26:04.002595 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:04.002573    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ebcf3a/rate-test-model-ebcf3a-kserve-6696dd5475-flt5m"]
Jul 29 01:26:04.224648 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:04.224615    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" path="/var/lib/kubelet/pods/add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6/volumes"
Jul 29 01:26:08.785508 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:08.785465    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:09.105293 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:09.105249    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:09.398975 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:09.398879    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:12.087340 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.087283    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:12.180988 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.180958    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:12.314656 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.314619    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"]
Jul 29 01:26:12.315069 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.315056    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" containerName="main"
Jul 29 01:26:12.315122 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.315071    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" containerName="main"
Jul 29 01:26:12.315164 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.315138    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="add2eb0f-c9c7-4f2e-bffa-c0dbbe31c6d6" containerName="main"
Jul 29 01:26:12.319767 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.319742    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.322878 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.322853    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e81b9d-serving-cert\""
Jul 29 01:26:12.327518 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.327482    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"]
Jul 29 01:26:12.360037 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.359941    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:12.469189 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.469157    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813b6e47-0f60-45c4-9cb7-b16d89590848-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.469369 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.469229    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813b6e47-0f60-45c4-9cb7-b16d89590848-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.469369 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.469265    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85cs2\" (UniqueName: \"kubernetes.io/projected/813b6e47-0f60-45c4-9cb7-b16d89590848-kube-api-access-85cs2\") pod \"maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.544768 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.544731    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:12.570711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.570677    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85cs2\" (UniqueName: \"kubernetes.io/projected/813b6e47-0f60-45c4-9cb7-b16d89590848-kube-api-access-85cs2\") pod \"maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.570905 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.570763    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813b6e47-0f60-45c4-9cb7-b16d89590848-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.570905 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.570799    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813b6e47-0f60-45c4-9cb7-b16d89590848-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.571421 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.571397    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813b6e47-0f60-45c4-9cb7-b16d89590848-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.573395 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.573377    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813b6e47-0f60-45c4-9cb7-b16d89590848-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.594981 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.594956    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85cs2\" (UniqueName: \"kubernetes.io/projected/813b6e47-0f60-45c4-9cb7-b16d89590848-kube-api-access-85cs2\") pod \"maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.634403 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.634326    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:12.786887 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.786857    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"]
Jul 29 01:26:12.788175 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:26:12.788143    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813b6e47_0f60_45c4_9cb7_b16d89590848.slice/crio-9f1cfb84ee6c7e3e5232a870a3e228ff197d801d7c4109f4f8e402dad9ae9bb1 WatchSource:0}: Error finding container 9f1cfb84ee6c7e3e5232a870a3e228ff197d801d7c4109f4f8e402dad9ae9bb1: Status 404 returned error can't find the container with id 9f1cfb84ee6c7e3e5232a870a3e228ff197d801d7c4109f4f8e402dad9ae9bb1
Jul 29 01:26:12.789736 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.789713    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:26:12.912643 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:12.912539    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:13.008893 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.008863    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:13.032887 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.032849    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd" event={"ID":"813b6e47-0f60-45c4-9cb7-b16d89590848","Type":"ContainerStarted","Data":"9f1cfb84ee6c7e3e5232a870a3e228ff197d801d7c4109f4f8e402dad9ae9bb1"}
Jul 29 01:26:13.055821 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.055784    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"]
Jul 29 01:26:13.060471 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.060452    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:13.065971 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.065947    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e81b9d\""
Jul 29 01:26:13.066406 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.066391    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e81b9d-dockercfg-cgsf6\""
Jul 29 01:26:13.075760 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.075737    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"]
Jul 29 01:26:13.176077 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.175985    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4fxg\" (UniqueName: \"kubernetes.io/projected/309ea4c9-7db9-4898-99fe-2240f8df97c9-kube-api-access-k4fxg\") pod \"payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz\" (UID: \"309ea4c9-7db9-4898-99fe-2240f8df97c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:13.176077 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.176036    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309ea4c9-7db9-4898-99fe-2240f8df97c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz\" (UID: \"309ea4c9-7db9-4898-99fe-2240f8df97c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:13.277513 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.277482    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4fxg\" (UniqueName: \"kubernetes.io/projected/309ea4c9-7db9-4898-99fe-2240f8df97c9-kube-api-access-k4fxg\") pod \"payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz\" (UID: \"309ea4c9-7db9-4898-99fe-2240f8df97c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:13.277587 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.277548    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309ea4c9-7db9-4898-99fe-2240f8df97c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz\" (UID: \"309ea4c9-7db9-4898-99fe-2240f8df97c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:13.278440 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.278414    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309ea4c9-7db9-4898-99fe-2240f8df97c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz\" (UID: \"309ea4c9-7db9-4898-99fe-2240f8df97c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:13.285452 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.285428    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4fxg\" (UniqueName: \"kubernetes.io/projected/309ea4c9-7db9-4898-99fe-2240f8df97c9-kube-api-access-k4fxg\") pod \"payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz\" (UID: \"309ea4c9-7db9-4898-99fe-2240f8df97c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:13.371997 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.371965    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:13.521531 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:13.521498    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"]
Jul 29 01:26:13.522800 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:26:13.522775    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309ea4c9_7db9_4898_99fe_2240f8df97c9.slice/crio-c633e6ff72d888467b00d3cf9a4fca7be50bb30d57dd211695c1499b987ec7d0 WatchSource:0}: Error finding container c633e6ff72d888467b00d3cf9a4fca7be50bb30d57dd211695c1499b987ec7d0: Status 404 returned error can't find the container with id c633e6ff72d888467b00d3cf9a4fca7be50bb30d57dd211695c1499b987ec7d0
Jul 29 01:26:14.041564 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:14.041524    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz" event={"ID":"309ea4c9-7db9-4898-99fe-2240f8df97c9","Type":"ContainerStarted","Data":"d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe"}
Jul 29 01:26:14.041767 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:14.041589    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:14.041767 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:14.041612    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz" event={"ID":"309ea4c9-7db9-4898-99fe-2240f8df97c9","Type":"ContainerStarted","Data":"c633e6ff72d888467b00d3cf9a4fca7be50bb30d57dd211695c1499b987ec7d0"}
Jul 29 01:26:14.042912 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:14.042888    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd" event={"ID":"813b6e47-0f60-45c4-9cb7-b16d89590848","Type":"ContainerStarted","Data":"9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51"}
Jul 29 01:26:14.043028 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:14.042929    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:14.060923 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:14.060880    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz" podStartSLOduration=1.0608666900000001 podStartE2EDuration="1.06086669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:26:14.058580379 +0000 UTC m=+3004.367508225" watchObservedRunningTime="2026-07-29 01:26:14.06086669 +0000 UTC m=+3004.369794532"
Jul 29 01:26:14.075815 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:14.075771    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd" podStartSLOduration=1.648165936 podStartE2EDuration="2.075757054s" totalImagesPullingTime="427.591118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:26:14.074911249 +0000 UTC m=+3004.383839089" watchObservedRunningTime="2026-07-29 01:26:14.075757054 +0000 UTC m=+3004.384684957"
Jul 29 01:26:20.053364 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:20.053332    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:26:25.050984 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:25.050952    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:26:37.145576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:37.145534    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:37.857399 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:37.857365    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:38.186135 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:38.186057    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:38.287209 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:38.287173    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:41.250386 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.250338    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:41.349048 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.349008    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:41.370756 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.370723    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"]
Jul 29 01:26:41.374700 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.374668    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:41.377663 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.377640    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7c9ee4-serving-cert\""
Jul 29 01:26:41.386224 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.386184    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"]
Jul 29 01:26:41.442150 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.442109    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:41.442150 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.442150    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdd4\" (UniqueName: \"kubernetes.io/projected/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-kube-api-access-khdd4\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:41.442416 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.442181    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:41.446531 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.446500    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:41.542853 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.542756    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:41.542853 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.542796    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdd4\" (UniqueName: \"kubernetes.io/projected/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-kube-api-access-khdd4\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:41.542853 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.542819    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:41.543162 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:26:41.542929    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-serving-cert: secret "maas-api-e2e-ipp-b-7c9ee4-serving-cert" not found
Jul 29 01:26:41.543162 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:26:41.543020    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-maas-api-tls podName:9e38ba91-4d39-45c1-9c2b-fe64eae278ae nodeName:}" failed. No retries permitted until 2026-07-29 01:26:42.043000661 +0000 UTC m=+3032.351928482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-maas-api-tls") pod "maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn" (UID: "9e38ba91-4d39-45c1-9c2b-fe64eae278ae") : secret "maas-api-e2e-ipp-b-7c9ee4-serving-cert" not found
Jul 29 01:26:41.543559 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.543540    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:41.553916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.553890    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdd4\" (UniqueName: \"kubernetes.io/projected/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-kube-api-access-khdd4\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:41.905181 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.905137    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:41.984001 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:41.983955    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:42.047796 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:42.047759    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:42.050322 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:42.050300    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:42.286947 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:42.286840    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:42.435628 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:42.435483    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"]
Jul 29 01:26:42.438044 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:26:42.438012    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e38ba91_4d39_45c1_9c2b_fe64eae278ae.slice/crio-78d9344fa23526947d6c76638ca589d27e8ef7aee3f239501b31c8c0898513ef WatchSource:0}: Error finding container 78d9344fa23526947d6c76638ca589d27e8ef7aee3f239501b31c8c0898513ef: Status 404 returned error can't find the container with id 78d9344fa23526947d6c76638ca589d27e8ef7aee3f239501b31c8c0898513ef
Jul 29 01:26:43.229995 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:43.229956    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn" event={"ID":"9e38ba91-4d39-45c1-9c2b-fe64eae278ae","Type":"ContainerStarted","Data":"b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2"}
Jul 29 01:26:43.230176 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:43.230016    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn" event={"ID":"9e38ba91-4d39-45c1-9c2b-fe64eae278ae","Type":"ContainerStarted","Data":"78d9344fa23526947d6c76638ca589d27e8ef7aee3f239501b31c8c0898513ef"}
Jul 29 01:26:43.230176 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:43.230072    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:26:43.249005 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:43.248946    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn" podStartSLOduration=1.898394683 podStartE2EDuration="2.248928142s" totalImagesPullingTime="350.533459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:26:43.246156086 +0000 UTC m=+3033.555083932" watchObservedRunningTime="2026-07-29 01:26:43.248928142 +0000 UTC m=+3033.557856032"
Jul 29 01:26:44.357492 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:44.357441    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:26:49.241140 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:26:49.241108    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:27:06.292061 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:06.291975    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:27:09.872118 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:09.872086    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:27:13.145576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:13.145542    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5cvtz_5a369ca0-d428-4bee-bc25-23a4ea6dd7fe/payload-processing/0.log"
Jul 29 01:27:19.250109 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:19.250073    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcfz7"]
Jul 29 01:27:19.254338 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:19.254314    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:19.264937 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:19.264912    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcfz7"]
Jul 29 01:27:19.298868 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:19.298832    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmnx7\" (UniqueName: \"kubernetes.io/projected/0b7832ea-3b7f-45e2-95cd-9e4d0756a82d-kube-api-access-kmnx7\") pod \"kuadrant-operator-catalog-dcfz7\" (UID: \"0b7832ea-3b7f-45e2-95cd-9e4d0756a82d\") " pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:19.399588 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:19.399547    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmnx7\" (UniqueName: \"kubernetes.io/projected/0b7832ea-3b7f-45e2-95cd-9e4d0756a82d-kube-api-access-kmnx7\") pod \"kuadrant-operator-catalog-dcfz7\" (UID: \"0b7832ea-3b7f-45e2-95cd-9e4d0756a82d\") " pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:19.408060 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:19.408023    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmnx7\" (UniqueName: \"kubernetes.io/projected/0b7832ea-3b7f-45e2-95cd-9e4d0756a82d-kube-api-access-kmnx7\") pod \"kuadrant-operator-catalog-dcfz7\" (UID: \"0b7832ea-3b7f-45e2-95cd-9e4d0756a82d\") " pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:19.567032 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:19.567000    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:19.707085 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:19.707058    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcfz7"]
Jul 29 01:27:20.457085 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:20.457044    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7" event={"ID":"0b7832ea-3b7f-45e2-95cd-9e4d0756a82d","Type":"ContainerStarted","Data":"baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc"}
Jul 29 01:27:20.457599 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:20.457105    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7" event={"ID":"0b7832ea-3b7f-45e2-95cd-9e4d0756a82d","Type":"ContainerStarted","Data":"5d443952f53bd8ecfca992048a0ceb8670cb9613c23b46977284a337ad5055f3"}
Jul 29 01:27:20.473024 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:20.472969    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7" podStartSLOduration=0.848782898 podStartE2EDuration="1.47295191s" totalImagesPullingTime="624.169012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:27:20.471926888 +0000 UTC m=+3070.780854731" watchObservedRunningTime="2026-07-29 01:27:20.47295191 +0000 UTC m=+3070.781879780"
Jul 29 01:27:25.956646 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:25.956609    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:27:29.568158 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:29.568110    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:29.568682 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:29.568180    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:29.590790 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:29.590756    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:30.536874 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:30.536846    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:30.585018 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:30.584969    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcfz7"]
Jul 29 01:27:32.526282 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:32.526234    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7" podUID="0b7832ea-3b7f-45e2-95cd-9e4d0756a82d" containerName="registry-server" containerID="cri-o://baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc" gracePeriod=2
Jul 29 01:27:32.783864 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:32.783803    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:32.923785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:32.923750    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmnx7\" (UniqueName: \"kubernetes.io/projected/0b7832ea-3b7f-45e2-95cd-9e4d0756a82d-kube-api-access-kmnx7\") pod \"0b7832ea-3b7f-45e2-95cd-9e4d0756a82d\" (UID: \"0b7832ea-3b7f-45e2-95cd-9e4d0756a82d\") "
Jul 29 01:27:32.925845 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:32.925816    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7832ea-3b7f-45e2-95cd-9e4d0756a82d-kube-api-access-kmnx7" (OuterVolumeSpecName: "kube-api-access-kmnx7") pod "0b7832ea-3b7f-45e2-95cd-9e4d0756a82d" (UID: "0b7832ea-3b7f-45e2-95cd-9e4d0756a82d"). InnerVolumeSpecName "kube-api-access-kmnx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:27:33.024644 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.024603    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmnx7\" (UniqueName: \"kubernetes.io/projected/0b7832ea-3b7f-45e2-95cd-9e4d0756a82d-kube-api-access-kmnx7\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:27:33.533774 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.533738    2586 generic.go:356] "Generic (PLEG): container finished" podID="0b7832ea-3b7f-45e2-95cd-9e4d0756a82d" containerID="baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc" exitCode=0
Jul 29 01:27:33.534175 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.533788    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7"
Jul 29 01:27:33.534175 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.533837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7" event={"ID":"0b7832ea-3b7f-45e2-95cd-9e4d0756a82d","Type":"ContainerDied","Data":"baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc"}
Jul 29 01:27:33.534175 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.533890    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dcfz7" event={"ID":"0b7832ea-3b7f-45e2-95cd-9e4d0756a82d","Type":"ContainerDied","Data":"5d443952f53bd8ecfca992048a0ceb8670cb9613c23b46977284a337ad5055f3"}
Jul 29 01:27:33.534175 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.533914    2586 scope.go:117] "RemoveContainer" containerID="baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc"
Jul 29 01:27:33.543213 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.543179    2586 scope.go:117] "RemoveContainer" containerID="baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc"
Jul 29 01:27:33.543490 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:27:33.543470    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc\": container with ID starting with baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc not found: ID does not exist" containerID="baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc"
Jul 29 01:27:33.543550 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.543504    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc"} err="failed to get container status \"baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc\": rpc error: code = NotFound desc = could not find container \"baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc\": container with ID starting with baddfe2714b871f57ba683b7db59acbb2b06cbcf1602de54a51f982423ab6adc not found: ID does not exist"
Jul 29 01:27:33.557408 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.557371    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcfz7"]
Jul 29 01:27:33.561357 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:33.561334    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcfz7"]
Jul 29 01:27:34.224506 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:34.224470    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7832ea-3b7f-45e2-95cd-9e4d0756a82d" path="/var/lib/kubelet/pods/0b7832ea-3b7f-45e2-95cd-9e4d0756a82d/volumes"
Jul 29 01:27:42.305916 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:42.305880    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:27:45.165761 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:45.165727    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:27:45.266414 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:27:45.266379    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:20.600997 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:20.600963    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz_309ea4c9-7db9-4898-99fe-2240f8df97c9/payload-processing/0.log"
Jul 29 01:28:20.721799 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:20.721769    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5cvtz_5a369ca0-d428-4bee-bc25-23a4ea6dd7fe/payload-processing/0.log"
Jul 29 01:28:21.125788 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:21.125757    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:21.417852 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:21.417767    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:21.675455 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:21.675370    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:24.727434 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:24.727399    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:24.831390 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:24.831358    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:24.962434 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:24.962399    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"]
Jul 29 01:28:24.962891 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:24.962873    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7832ea-3b7f-45e2-95cd-9e4d0756a82d" containerName="registry-server"
Jul 29 01:28:24.962891 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:24.962892    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7832ea-3b7f-45e2-95cd-9e4d0756a82d" containerName="registry-server"
Jul 29 01:28:24.963032 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:24.962966    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7832ea-3b7f-45e2-95cd-9e4d0756a82d" containerName="registry-server"
Jul 29 01:28:24.966318 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:24.966290    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:24.970015 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:24.969990    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-06c978-serving-cert\""
Jul 29 01:28:24.985129 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:24.985061    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"]
Jul 29 01:28:25.008766 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.008734    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:25.120115 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.120078    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.120345 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.120136    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.120345 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.120273    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6m6\" (UniqueName: \"kubernetes.io/projected/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-kube-api-access-fn6m6\") pod \"maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.221116 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.221082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.221340 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.221129    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.221340 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.221165    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6m6\" (UniqueName: \"kubernetes.io/projected/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-kube-api-access-fn6m6\") pod \"maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.221830 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.221802    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.223643 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.223620    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.231716 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.231693    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6m6\" (UniqueName: \"kubernetes.io/projected/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-kube-api-access-fn6m6\") pod \"maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.235470 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.235409    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:25.277317 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.277275    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:25.433153 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.433117    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"]
Jul 29 01:28:25.434434 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:28:25.434406    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb6ff93_cf7e_49c3_bd3f_36d53ef567bb.slice/crio-e222ff3b87952076ea2926e95e9c38c6f1753cafa8defc42eddc687fb74f27e6 WatchSource:0}: Error finding container e222ff3b87952076ea2926e95e9c38c6f1753cafa8defc42eddc687fb74f27e6: Status 404 returned error can't find the container with id e222ff3b87952076ea2926e95e9c38c6f1753cafa8defc42eddc687fb74f27e6
Jul 29 01:28:25.552713 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.552622    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:25.710323 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.710241    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:25.765606 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.765573    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"]
Jul 29 01:28:25.770663 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.770641    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:25.780585 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.779685    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-06c978-dockercfg-ltgc7\""
Jul 29 01:28:25.780585 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.779745    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-06c978\""
Jul 29 01:28:25.790401 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.790358    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"]
Jul 29 01:28:25.841670 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.841629    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz" event={"ID":"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb","Type":"ContainerStarted","Data":"e222ff3b87952076ea2926e95e9c38c6f1753cafa8defc42eddc687fb74f27e6"}
Jul 29 01:28:25.928858 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.928814    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz\" (UID: \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:25.929053 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:25.928882    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t82hb\" (UniqueName: \"kubernetes.io/projected/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-kube-api-access-t82hb\") pod \"payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz\" (UID: \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:26.030334 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.030302    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz\" (UID: \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:26.030485 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.030354    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t82hb\" (UniqueName: \"kubernetes.io/projected/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-kube-api-access-t82hb\") pod \"payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz\" (UID: \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:26.031084 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.031058    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz\" (UID: \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:26.039296 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.039263    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t82hb\" (UniqueName: \"kubernetes.io/projected/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-kube-api-access-t82hb\") pod \"payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz\" (UID: \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:26.087456 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.087379    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:26.251530 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.251493    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"]
Jul 29 01:28:26.252814 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:28:26.252785    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cd2c32_dc17_42c8_9b9c_847300e2f2a4.slice/crio-26a2534178f8ab52207d25195fb11383a3952ec4f35dd106818769ba390cd25e WatchSource:0}: Error finding container 26a2534178f8ab52207d25195fb11383a3952ec4f35dd106818769ba390cd25e: Status 404 returned error can't find the container with id 26a2534178f8ab52207d25195fb11383a3952ec4f35dd106818769ba390cd25e
Jul 29 01:28:26.848857 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.848799    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz" event={"ID":"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb","Type":"ContainerStarted","Data":"cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0"}
Jul 29 01:28:26.849378 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.848883    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:26.850511 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.850485    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz" event={"ID":"37cd2c32-dc17-42c8-9b9c-847300e2f2a4","Type":"ContainerStarted","Data":"64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3"}
Jul 29 01:28:26.850617 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.850527    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz" event={"ID":"37cd2c32-dc17-42c8-9b9c-847300e2f2a4","Type":"ContainerStarted","Data":"26a2534178f8ab52207d25195fb11383a3952ec4f35dd106818769ba390cd25e"}
Jul 29 01:28:26.850617 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.850549    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:26.869142 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.869081    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz" podStartSLOduration=2.350297722 podStartE2EDuration="2.869063109s" totalImagesPullingTime="518.765387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:28:26.867051441 +0000 UTC m=+3137.175979287" watchObservedRunningTime="2026-07-29 01:28:26.869063109 +0000 UTC m=+3137.177990978"
Jul 29 01:28:26.887647 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:26.887588    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz" podStartSLOduration=1.8875682999999999 podStartE2EDuration="1.8875683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:28:26.885429525 +0000 UTC m=+3137.194357368" watchObservedRunningTime="2026-07-29 01:28:26.8875683 +0000 UTC m=+3137.196496144"
Jul 29 01:28:32.863722 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:32.863640    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:37.860296 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:37.860266    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:49.722492 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:49.722453    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:55.589435 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.589401    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"]
Jul 29 01:28:55.590360 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.590289    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz" podUID="feb6ff93-cf7e-49c3-bd3f-36d53ef567bb" containerName="maas-api" containerID="cri-o://cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0" gracePeriod=30
Jul 29 01:28:55.617528 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.617497    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:55.660941 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.660900    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"]
Jul 29 01:28:55.661387 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.661263    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz" podUID="37cd2c32-dc17-42c8-9b9c-847300e2f2a4" containerName="payload-pre-processing" containerID="cri-o://64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3" gracePeriod=30
Jul 29 01:28:55.718672 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.718485    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:55.784461 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.784426    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:55.845257 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.843852    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:55.925481 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.925457    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:55.959545 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:55.959516    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:56.002772 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.002727    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-plugins-config-volume\") pod \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\" (UID: \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\") "
Jul 29 01:28:56.002980 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.002785    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn6m6\" (UniqueName: \"kubernetes.io/projected/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-kube-api-access-fn6m6\") pod \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") "
Jul 29 01:28:56.002980 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.002841    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t82hb\" (UniqueName: \"kubernetes.io/projected/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-kube-api-access-t82hb\") pod \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\" (UID: \"37cd2c32-dc17-42c8-9b9c-847300e2f2a4\") "
Jul 29 01:28:56.002980 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.002864    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-openshift-service-ca\") pod \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") "
Jul 29 01:28:56.002980 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.002890    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-maas-api-tls\") pod \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\" (UID: \"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb\") "
Jul 29 01:28:56.003312 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.003283    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "feb6ff93-cf7e-49c3-bd3f-36d53ef567bb" (UID: "feb6ff93-cf7e-49c3-bd3f-36d53ef567bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:28:56.003391 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.003314    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37cd2c32-dc17-42c8-9b9c-847300e2f2a4" (UID: "37cd2c32-dc17-42c8-9b9c-847300e2f2a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:28:56.004988 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.004960    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "feb6ff93-cf7e-49c3-bd3f-36d53ef567bb" (UID: "feb6ff93-cf7e-49c3-bd3f-36d53ef567bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:28:56.005105 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.005015    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-kube-api-access-t82hb" (OuterVolumeSpecName: "kube-api-access-t82hb") pod "37cd2c32-dc17-42c8-9b9c-847300e2f2a4" (UID: "37cd2c32-dc17-42c8-9b9c-847300e2f2a4"). InnerVolumeSpecName "kube-api-access-t82hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:28:56.005105 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.005042    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-kube-api-access-fn6m6" (OuterVolumeSpecName: "kube-api-access-fn6m6") pod "feb6ff93-cf7e-49c3-bd3f-36d53ef567bb" (UID: "feb6ff93-cf7e-49c3-bd3f-36d53ef567bb"). InnerVolumeSpecName "kube-api-access-fn6m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:28:56.030292 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.030255    2586 generic.go:356] "Generic (PLEG): container finished" podID="feb6ff93-cf7e-49c3-bd3f-36d53ef567bb" containerID="cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0" exitCode=0
Jul 29 01:28:56.030472 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.030299    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"
Jul 29 01:28:56.030472 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.030340    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz" event={"ID":"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb","Type":"ContainerDied","Data":"cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0"}
Jul 29 01:28:56.030472 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.030414    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz" event={"ID":"feb6ff93-cf7e-49c3-bd3f-36d53ef567bb","Type":"ContainerDied","Data":"e222ff3b87952076ea2926e95e9c38c6f1753cafa8defc42eddc687fb74f27e6"}
Jul 29 01:28:56.030472 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.030442    2586 scope.go:117] "RemoveContainer" containerID="cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0"
Jul 29 01:28:56.031804 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.031780    2586 generic.go:356] "Generic (PLEG): container finished" podID="37cd2c32-dc17-42c8-9b9c-847300e2f2a4" containerID="64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3" exitCode=0
Jul 29 01:28:56.031973 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.031821    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"
Jul 29 01:28:56.031973 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.031862    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz" event={"ID":"37cd2c32-dc17-42c8-9b9c-847300e2f2a4","Type":"ContainerDied","Data":"64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3"}
Jul 29 01:28:56.031973 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.031914    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz" event={"ID":"37cd2c32-dc17-42c8-9b9c-847300e2f2a4","Type":"ContainerDied","Data":"26a2534178f8ab52207d25195fb11383a3952ec4f35dd106818769ba390cd25e"}
Jul 29 01:28:56.041187 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.041169    2586 scope.go:117] "RemoveContainer" containerID="cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0"
Jul 29 01:28:56.041491 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:28:56.041466    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0\": container with ID starting with cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0 not found: ID does not exist" containerID="cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0"
Jul 29 01:28:56.041548 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.041504    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0"} err="failed to get container status \"cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0\": rpc error: code = NotFound desc = could not find container \"cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0\": container with ID starting with cdfaca525569bf1f90870a0e4dd769a8b97facedcd186bc12eba39bf36c1a7b0 not found: ID does not exist"
Jul 29 01:28:56.041548 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.041532    2586 scope.go:117] "RemoveContainer" containerID="64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3"
Jul 29 01:28:56.051081 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.051062    2586 scope.go:117] "RemoveContainer" containerID="64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3"
Jul 29 01:28:56.051364 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:28:56.051344    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3\": container with ID starting with 64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3 not found: ID does not exist" containerID="64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3"
Jul 29 01:28:56.051423 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.051372    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3"} err="failed to get container status \"64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3\": rpc error: code = NotFound desc = could not find container \"64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3\": container with ID starting with 64ec5b5d27b6d983bda1b7d59816d6ae5f2055259cd9506e1eed85d5fd9e4ba3 not found: ID does not exist"
Jul 29 01:28:56.058395 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.058372    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"]
Jul 29 01:28:56.062670 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.062645    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06c978-64dd5b654d-pkjgz"]
Jul 29 01:28:56.073622 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.073593    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"]
Jul 29 01:28:56.079774 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.079744    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06c978-5c9589b5d7-fk9pz"]
Jul 29 01:28:56.104640 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.104564    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:28:56.104640 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.104592    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn6m6\" (UniqueName: \"kubernetes.io/projected/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-kube-api-access-fn6m6\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:28:56.104640 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.104602    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t82hb\" (UniqueName: \"kubernetes.io/projected/37cd2c32-dc17-42c8-9b9c-847300e2f2a4-kube-api-access-t82hb\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:28:56.104640 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.104611    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:28:56.104640 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.104621    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:28:56.164936 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.164903    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:28:56.222142 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.222111    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37cd2c32-dc17-42c8-9b9c-847300e2f2a4" path="/var/lib/kubelet/pods/37cd2c32-dc17-42c8-9b9c-847300e2f2a4/volumes"
Jul 29 01:28:56.222572 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:28:56.222557    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb6ff93-cf7e-49c3-bd3f-36d53ef567bb" path="/var/lib/kubelet/pods/feb6ff93-cf7e-49c3-bd3f-36d53ef567bb/volumes"
Jul 29 01:29:01.830928 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:01.830892    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:29:01.930941 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:01.930911    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:29:02.086864 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:02.086719    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:29:02.309936 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:02.309903    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:29:21.085915 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.085811    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"]
Jul 29 01:29:21.086502 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.086087    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd" podUID="813b6e47-0f60-45c4-9cb7-b16d89590848" containerName="maas-api" containerID="cri-o://9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51" gracePeriod=30
Jul 29 01:29:21.113138 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.113110    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:29:21.141703 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.141666    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"]
Jul 29 01:29:21.141930 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.141902    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz" podUID="309ea4c9-7db9-4898-99fe-2240f8df97c9" containerName="payload-processing" containerID="cri-o://d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe" gracePeriod=30
Jul 29 01:29:21.403464 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.403433    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:29:21.436450 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.436426    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:29:21.548153 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.548118    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85cs2\" (UniqueName: \"kubernetes.io/projected/813b6e47-0f60-45c4-9cb7-b16d89590848-kube-api-access-85cs2\") pod \"813b6e47-0f60-45c4-9cb7-b16d89590848\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") "
Jul 29 01:29:21.548365 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.548243    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309ea4c9-7db9-4898-99fe-2240f8df97c9-plugins-config-volume\") pod \"309ea4c9-7db9-4898-99fe-2240f8df97c9\" (UID: \"309ea4c9-7db9-4898-99fe-2240f8df97c9\") "
Jul 29 01:29:21.548365 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.548339    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813b6e47-0f60-45c4-9cb7-b16d89590848-maas-api-tls\") pod \"813b6e47-0f60-45c4-9cb7-b16d89590848\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") "
Jul 29 01:29:21.548489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.548371    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4fxg\" (UniqueName: \"kubernetes.io/projected/309ea4c9-7db9-4898-99fe-2240f8df97c9-kube-api-access-k4fxg\") pod \"309ea4c9-7db9-4898-99fe-2240f8df97c9\" (UID: \"309ea4c9-7db9-4898-99fe-2240f8df97c9\") "
Jul 29 01:29:21.548489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.548419    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813b6e47-0f60-45c4-9cb7-b16d89590848-openshift-service-ca\") pod \"813b6e47-0f60-45c4-9cb7-b16d89590848\" (UID: \"813b6e47-0f60-45c4-9cb7-b16d89590848\") "
Jul 29 01:29:21.548780 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.548749    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309ea4c9-7db9-4898-99fe-2240f8df97c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "309ea4c9-7db9-4898-99fe-2240f8df97c9" (UID: "309ea4c9-7db9-4898-99fe-2240f8df97c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:29:21.548891 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.548818    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813b6e47-0f60-45c4-9cb7-b16d89590848-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "813b6e47-0f60-45c4-9cb7-b16d89590848" (UID: "813b6e47-0f60-45c4-9cb7-b16d89590848"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:29:21.550487 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.550457    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813b6e47-0f60-45c4-9cb7-b16d89590848-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "813b6e47-0f60-45c4-9cb7-b16d89590848" (UID: "813b6e47-0f60-45c4-9cb7-b16d89590848"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:29:21.550487 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.550463    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309ea4c9-7db9-4898-99fe-2240f8df97c9-kube-api-access-k4fxg" (OuterVolumeSpecName: "kube-api-access-k4fxg") pod "309ea4c9-7db9-4898-99fe-2240f8df97c9" (UID: "309ea4c9-7db9-4898-99fe-2240f8df97c9"). InnerVolumeSpecName "kube-api-access-k4fxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:29:21.550632 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.550486    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813b6e47-0f60-45c4-9cb7-b16d89590848-kube-api-access-85cs2" (OuterVolumeSpecName: "kube-api-access-85cs2") pod "813b6e47-0f60-45c4-9cb7-b16d89590848" (UID: "813b6e47-0f60-45c4-9cb7-b16d89590848"). InnerVolumeSpecName "kube-api-access-85cs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:29:21.649646 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.649544    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813b6e47-0f60-45c4-9cb7-b16d89590848-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:29:21.649646 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.649582    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4fxg\" (UniqueName: \"kubernetes.io/projected/309ea4c9-7db9-4898-99fe-2240f8df97c9-kube-api-access-k4fxg\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:29:21.649646 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.649593    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813b6e47-0f60-45c4-9cb7-b16d89590848-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:29:21.649646 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.649604    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85cs2\" (UniqueName: \"kubernetes.io/projected/813b6e47-0f60-45c4-9cb7-b16d89590848-kube-api-access-85cs2\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:29:21.649646 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:21.649613    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309ea4c9-7db9-4898-99fe-2240f8df97c9-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:29:22.202739 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.202702    2586 generic.go:356] "Generic (PLEG): container finished" podID="813b6e47-0f60-45c4-9cb7-b16d89590848" containerID="9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51" exitCode=0
Jul 29 01:29:22.203219 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.202750    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"
Jul 29 01:29:22.203219 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.202791    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd" event={"ID":"813b6e47-0f60-45c4-9cb7-b16d89590848","Type":"ContainerDied","Data":"9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51"}
Jul 29 01:29:22.203219 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.202847    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd" event={"ID":"813b6e47-0f60-45c4-9cb7-b16d89590848","Type":"ContainerDied","Data":"9f1cfb84ee6c7e3e5232a870a3e228ff197d801d7c4109f4f8e402dad9ae9bb1"}
Jul 29 01:29:22.203219 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.202874    2586 scope.go:117] "RemoveContainer" containerID="9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51"
Jul 29 01:29:22.204487 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.204458    2586 generic.go:356] "Generic (PLEG): container finished" podID="309ea4c9-7db9-4898-99fe-2240f8df97c9" containerID="d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe" exitCode=0
Jul 29 01:29:22.204584 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.204505    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"
Jul 29 01:29:22.204584 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.204516    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz" event={"ID":"309ea4c9-7db9-4898-99fe-2240f8df97c9","Type":"ContainerDied","Data":"d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe"}
Jul 29 01:29:22.204584 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.204555    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz" event={"ID":"309ea4c9-7db9-4898-99fe-2240f8df97c9","Type":"ContainerDied","Data":"c633e6ff72d888467b00d3cf9a4fca7be50bb30d57dd211695c1499b987ec7d0"}
Jul 29 01:29:22.219835 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.219132    2586 scope.go:117] "RemoveContainer" containerID="9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51"
Jul 29 01:29:22.219835 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:29:22.219647    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51\": container with ID starting with 9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51 not found: ID does not exist" containerID="9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51"
Jul 29 01:29:22.219835 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.219680    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51"} err="failed to get container status \"9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51\": rpc error: code = NotFound desc = could not find container \"9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51\": container with ID starting with 9e39efdede08b5de9d1be5bab6a45c3d0eb70446e49274f380d32894339f1c51 not found: ID does not exist"
Jul 29 01:29:22.219835 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.219705    2586 scope.go:117] "RemoveContainer" containerID="d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe"
Jul 29 01:29:22.230898 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.230868    2586 scope.go:117] "RemoveContainer" containerID="d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe"
Jul 29 01:29:22.231676 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:29:22.231648    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe\": container with ID starting with d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe not found: ID does not exist" containerID="d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe"
Jul 29 01:29:22.231771 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.231682    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe"} err="failed to get container status \"d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe\": rpc error: code = NotFound desc = could not find container \"d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe\": container with ID starting with d01143eb76596f9babe4ce2b6c4a13e45c9288468e11b44d7de4a2a486121bfe not found: ID does not exist"
Jul 29 01:29:22.234258 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.234232    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"]
Jul 29 01:29:22.237660 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.237632    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e81b9d-9878f694d-kqxhd"]
Jul 29 01:29:22.250548 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.250522    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"]
Jul 29 01:29:22.256527 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.256502    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e81b9d-b4888b864-2tqsz"]
Jul 29 01:29:22.296617 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:22.296583    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:29:24.221493 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:24.221456    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="309ea4c9-7db9-4898-99fe-2240f8df97c9" path="/var/lib/kubelet/pods/309ea4c9-7db9-4898-99fe-2240f8df97c9/volumes"
Jul 29 01:29:24.221881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:24.221855    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="813b6e47-0f60-45c4-9cb7-b16d89590848" path="/var/lib/kubelet/pods/813b6e47-0f60-45c4-9cb7-b16d89590848/volumes"
Jul 29 01:29:26.738939 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:29:26.738899    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:00.138469 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.138395    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9"]
Jul 29 01:30:00.139178 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139156    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cd2c32-dc17-42c8-9b9c-847300e2f2a4" containerName="payload-pre-processing"
Jul 29 01:30:00.139290 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139182    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cd2c32-dc17-42c8-9b9c-847300e2f2a4" containerName="payload-pre-processing"
Jul 29 01:30:00.139290 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139213    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb6ff93-cf7e-49c3-bd3f-36d53ef567bb" containerName="maas-api"
Jul 29 01:30:00.139290 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139223    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb6ff93-cf7e-49c3-bd3f-36d53ef567bb" containerName="maas-api"
Jul 29 01:30:00.139290 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139249    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309ea4c9-7db9-4898-99fe-2240f8df97c9" containerName="payload-processing"
Jul 29 01:30:00.139290 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139258    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="309ea4c9-7db9-4898-99fe-2240f8df97c9" containerName="payload-processing"
Jul 29 01:30:00.139290 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139275    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813b6e47-0f60-45c4-9cb7-b16d89590848" containerName="maas-api"
Jul 29 01:30:00.139290 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139283    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="813b6e47-0f60-45c4-9cb7-b16d89590848" containerName="maas-api"
Jul 29 01:30:00.139625 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139396    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb6ff93-cf7e-49c3-bd3f-36d53ef567bb" containerName="maas-api"
Jul 29 01:30:00.139625 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139410    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="309ea4c9-7db9-4898-99fe-2240f8df97c9" containerName="payload-processing"
Jul 29 01:30:00.139625 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139422    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cd2c32-dc17-42c8-9b9c-847300e2f2a4" containerName="payload-pre-processing"
Jul 29 01:30:00.139625 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.139432    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="813b6e47-0f60-45c4-9cb7-b16d89590848" containerName="maas-api"
Jul 29 01:30:00.143278 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.143253    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm"]
Jul 29 01:30:00.143401 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.143365    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9"
Jul 29 01:30:00.145941 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.145905    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:30:00.147330 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.147299    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm"
Jul 29 01:30:00.150442 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.150418    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9"]
Jul 29 01:30:00.154688 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.154663    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm"]
Jul 29 01:30:00.159477 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.159419    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9"
Jul 29 01:30:00.166445 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.166425    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm"
Jul 29 01:30:00.326570 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.326538    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9"]
Jul 29 01:30:00.329394 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:30:00.329358    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8537d4_3247_488c_8234_7b79c358055d.slice/crio-31b62fc5fa29d6bbfc483af91a3e27a112cbc6c54c2b5951b07851f18f4dfe00 WatchSource:0}: Error finding container 31b62fc5fa29d6bbfc483af91a3e27a112cbc6c54c2b5951b07851f18f4dfe00: Status 404 returned error can't find the container with id 31b62fc5fa29d6bbfc483af91a3e27a112cbc6c54c2b5951b07851f18f4dfe00
Jul 29 01:30:00.364728 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.364697    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm"]
Jul 29 01:30:00.372536 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:30:00.372505    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8e095c_3646_42d8_bc69_1e841e2aa0a1.slice/crio-8afab99882e252ef430a895cef827841c1b19cf1572d1a81a80fd63115b6b437 WatchSource:0}: Error finding container 8afab99882e252ef430a895cef827841c1b19cf1572d1a81a80fd63115b6b437: Status 404 returned error can't find the container with id 8afab99882e252ef430a895cef827841c1b19cf1572d1a81a80fd63115b6b437
Jul 29 01:30:00.429760 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.429715    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm" event={"ID":"ef8e095c-3646-42d8-bc69-1e841e2aa0a1","Type":"ContainerStarted","Data":"8afab99882e252ef430a895cef827841c1b19cf1572d1a81a80fd63115b6b437"}
Jul 29 01:30:00.431051 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:00.431023    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9" event={"ID":"0d8537d4-3247-488c-8234-7b79c358055d","Type":"ContainerStarted","Data":"31b62fc5fa29d6bbfc483af91a3e27a112cbc6c54c2b5951b07851f18f4dfe00"}
Jul 29 01:30:01.438423 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:01.438380    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm" event={"ID":"ef8e095c-3646-42d8-bc69-1e841e2aa0a1","Type":"ContainerStarted","Data":"5e61bc19557e20b1507bbef02c72262be227a94fd084e040855db6d5ca4ef6e4"}
Jul 29 01:30:01.439914 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:01.439887    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9" event={"ID":"0d8537d4-3247-488c-8234-7b79c358055d","Type":"ContainerStarted","Data":"50e53b2c8d6642ce7c06dac4a463aac8c3bc705e4053616ebe2787edbf55573f"}
Jul 29 01:30:01.453650 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:01.453596    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm" podStartSLOduration=1.453582222 podStartE2EDuration="1.453582222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:30:01.453256314 +0000 UTC m=+3231.762184158" watchObservedRunningTime="2026-07-29 01:30:01.453582222 +0000 UTC m=+3231.762510063"
Jul 29 01:30:01.469085 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:01.469031    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9" podStartSLOduration=1.469010151 podStartE2EDuration="1.469010151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:30:01.467599286 +0000 UTC m=+3231.776527129" watchObservedRunningTime="2026-07-29 01:30:01.469010151 +0000 UTC m=+3231.777937994"
Jul 29 01:30:02.447468 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:02.447426    2586 generic.go:356] "Generic (PLEG): container finished" podID="0d8537d4-3247-488c-8234-7b79c358055d" containerID="50e53b2c8d6642ce7c06dac4a463aac8c3bc705e4053616ebe2787edbf55573f" exitCode=0
Jul 29 01:30:02.447874 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:02.447511    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9" event={"ID":"0d8537d4-3247-488c-8234-7b79c358055d","Type":"ContainerDied","Data":"50e53b2c8d6642ce7c06dac4a463aac8c3bc705e4053616ebe2787edbf55573f"}
Jul 29 01:30:02.449046 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:02.449025    2586 generic.go:356] "Generic (PLEG): container finished" podID="ef8e095c-3646-42d8-bc69-1e841e2aa0a1" containerID="5e61bc19557e20b1507bbef02c72262be227a94fd084e040855db6d5ca4ef6e4" exitCode=0
Jul 29 01:30:02.449146 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:02.449054    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm" event={"ID":"ef8e095c-3646-42d8-bc69-1e841e2aa0a1","Type":"ContainerDied","Data":"5e61bc19557e20b1507bbef02c72262be227a94fd084e040855db6d5ca4ef6e4"}
Jul 29 01:30:03.644420 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:03.644386    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm"
Jul 29 01:30:03.647517 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:03.647497    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9"
Jul 29 01:30:04.300040 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:04.300011    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:04.369725 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:04.369693    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:04.464135 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:04.464106    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9"
Jul 29 01:30:04.464346 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:04.464143    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9" event={"ID":"0d8537d4-3247-488c-8234-7b79c358055d","Type":"ContainerDied","Data":"31b62fc5fa29d6bbfc483af91a3e27a112cbc6c54c2b5951b07851f18f4dfe00"}
Jul 29 01:30:04.464346 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:04.464218    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31b62fc5fa29d6bbfc483af91a3e27a112cbc6c54c2b5951b07851f18f4dfe00"
Jul 29 01:30:04.465674 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:04.465644    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm" event={"ID":"ef8e095c-3646-42d8-bc69-1e841e2aa0a1","Type":"ContainerDied","Data":"8afab99882e252ef430a895cef827841c1b19cf1572d1a81a80fd63115b6b437"}
Jul 29 01:30:04.465813 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:04.465682    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm"
Jul 29 01:30:04.465813 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:04.465699    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8afab99882e252ef430a895cef827841c1b19cf1572d1a81a80fd63115b6b437"
Jul 29 01:30:09.900859 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:09.900824    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:10.009410 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.009373    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:10.202807 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.202692    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:10.430826 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.430794    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:10.548269 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.548160    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"]
Jul 29 01:30:10.548711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.548692    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8537d4-3247-488c-8234-7b79c358055d" containerName="cleanup"
Jul 29 01:30:10.548797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.548714    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8537d4-3247-488c-8234-7b79c358055d" containerName="cleanup"
Jul 29 01:30:10.548797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.548728    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8e095c-3646-42d8-bc69-1e841e2aa0a1" containerName="cleanup"
Jul 29 01:30:10.548797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.548737    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8e095c-3646-42d8-bc69-1e841e2aa0a1" containerName="cleanup"
Jul 29 01:30:10.548939 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.548833    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8e095c-3646-42d8-bc69-1e841e2aa0a1" containerName="cleanup"
Jul 29 01:30:10.548939 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.548849    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8537d4-3247-488c-8234-7b79c358055d" containerName="cleanup"
Jul 29 01:30:10.552187 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.552164    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"
Jul 29 01:30:10.556737 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.556714    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:30:10.563364 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.563338    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"]
Jul 29 01:30:10.707496 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.707462    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4859dbf2-9100-4493-b672-3e281fac1eb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw\" (UID: \"4859dbf2-9100-4493-b672-3e281fac1eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"
Jul 29 01:30:10.808226 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.808103    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4859dbf2-9100-4493-b672-3e281fac1eb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw\" (UID: \"4859dbf2-9100-4493-b672-3e281fac1eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"
Jul 29 01:30:10.808833 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.808815    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4859dbf2-9100-4493-b672-3e281fac1eb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw\" (UID: \"4859dbf2-9100-4493-b672-3e281fac1eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"
Jul 29 01:30:10.866528 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:10.866495    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"
Jul 29 01:30:11.021599 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:11.021560    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"]
Jul 29 01:30:11.022634 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:30:11.022607    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4859dbf2_9100_4493_b672_3e281fac1eb2.slice/crio-588f01fb04423600815d613827ecadadf09481b26bd5b983eecace4a2e735b3a WatchSource:0}: Error finding container 588f01fb04423600815d613827ecadadf09481b26bd5b983eecace4a2e735b3a: Status 404 returned error can't find the container with id 588f01fb04423600815d613827ecadadf09481b26bd5b983eecace4a2e735b3a
Jul 29 01:30:11.511227 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:11.511173    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw" event={"ID":"4859dbf2-9100-4493-b672-3e281fac1eb2","Type":"ContainerStarted","Data":"52e0cf5005f64507aa8af7a9262a70a546369fa477c7c4c0d7fe72ca681bbbcb"}
Jul 29 01:30:11.511435 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:11.511253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw" event={"ID":"4859dbf2-9100-4493-b672-3e281fac1eb2","Type":"ContainerStarted","Data":"588f01fb04423600815d613827ecadadf09481b26bd5b983eecace4a2e735b3a"}
Jul 29 01:30:11.529480 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:11.529414    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw" podStartSLOduration=1.529398617 podStartE2EDuration="1.529398617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:30:11.526354748 +0000 UTC m=+3241.835282607" watchObservedRunningTime="2026-07-29 01:30:11.529398617 +0000 UTC m=+3241.838326459"
Jul 29 01:30:12.518891 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:12.518862    2586 generic.go:356] "Generic (PLEG): container finished" podID="4859dbf2-9100-4493-b672-3e281fac1eb2" containerID="52e0cf5005f64507aa8af7a9262a70a546369fa477c7c4c0d7fe72ca681bbbcb" exitCode=0
Jul 29 01:30:12.519220 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:12.518933    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw" event={"ID":"4859dbf2-9100-4493-b672-3e281fac1eb2","Type":"ContainerDied","Data":"52e0cf5005f64507aa8af7a9262a70a546369fa477c7c4c0d7fe72ca681bbbcb"}
Jul 29 01:30:13.668677 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:13.668643    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"
Jul 29 01:30:13.840951 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:13.840919    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4859dbf2-9100-4493-b672-3e281fac1eb2-maas-api-service-ca\") pod \"4859dbf2-9100-4493-b672-3e281fac1eb2\" (UID: \"4859dbf2-9100-4493-b672-3e281fac1eb2\") "
Jul 29 01:30:13.841336 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:13.841309    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4859dbf2-9100-4493-b672-3e281fac1eb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4859dbf2-9100-4493-b672-3e281fac1eb2" (UID: "4859dbf2-9100-4493-b672-3e281fac1eb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:30:13.941711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:13.941668    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4859dbf2-9100-4493-b672-3e281fac1eb2-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:14.533860 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:14.533778    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"
Jul 29 01:30:14.533997 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:14.533767    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw" event={"ID":"4859dbf2-9100-4493-b672-3e281fac1eb2","Type":"ContainerDied","Data":"588f01fb04423600815d613827ecadadf09481b26bd5b983eecace4a2e735b3a"}
Jul 29 01:30:14.533997 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:14.533903    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="588f01fb04423600815d613827ecadadf09481b26bd5b983eecace4a2e735b3a"
Jul 29 01:30:21.048943 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.048905    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"]
Jul 29 01:30:21.049417 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.049150    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn" podUID="9e38ba91-4d39-45c1-9c2b-fe64eae278ae" containerName="maas-api" containerID="cri-o://b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2" gracePeriod=30
Jul 29 01:30:21.081432 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.081396    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:21.177832 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.177799    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:21.245905 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.245725    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"]
Jul 29 01:30:21.248552 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.248501    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:21.251066 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.251043    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c9ee4-nncfw"]
Jul 29 01:30:21.310238 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.310212    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:30:21.312730 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.312707    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:21.399131 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.399090    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khdd4\" (UniqueName: \"kubernetes.io/projected/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-kube-api-access-khdd4\") pod \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") "
Jul 29 01:30:21.399131 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.399132    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-openshift-service-ca\") pod \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") "
Jul 29 01:30:21.399406 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.399160    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-maas-api-tls\") pod \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\" (UID: \"9e38ba91-4d39-45c1-9c2b-fe64eae278ae\") "
Jul 29 01:30:21.399619 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.399593    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e38ba91-4d39-45c1-9c2b-fe64eae278ae" (UID: "9e38ba91-4d39-45c1-9c2b-fe64eae278ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:30:21.401436 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.401401    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-kube-api-access-khdd4" (OuterVolumeSpecName: "kube-api-access-khdd4") pod "9e38ba91-4d39-45c1-9c2b-fe64eae278ae" (UID: "9e38ba91-4d39-45c1-9c2b-fe64eae278ae"). InnerVolumeSpecName "kube-api-access-khdd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:30:21.401534 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.401428    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e38ba91-4d39-45c1-9c2b-fe64eae278ae" (UID: "9e38ba91-4d39-45c1-9c2b-fe64eae278ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:30:21.500521 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.500478    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khdd4\" (UniqueName: \"kubernetes.io/projected/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-kube-api-access-khdd4\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:21.500521 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.500512    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:21.500521 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.500525    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e38ba91-4d39-45c1-9c2b-fe64eae278ae-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:21.577912 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.577818    2586 generic.go:356] "Generic (PLEG): container finished" podID="9e38ba91-4d39-45c1-9c2b-fe64eae278ae" containerID="b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2" exitCode=0
Jul 29 01:30:21.577912 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.577869    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"
Jul 29 01:30:21.577912 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.577887    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn" event={"ID":"9e38ba91-4d39-45c1-9c2b-fe64eae278ae","Type":"ContainerDied","Data":"b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2"}
Jul 29 01:30:21.578190 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.577981    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn" event={"ID":"9e38ba91-4d39-45c1-9c2b-fe64eae278ae","Type":"ContainerDied","Data":"78d9344fa23526947d6c76638ca589d27e8ef7aee3f239501b31c8c0898513ef"}
Jul 29 01:30:21.578190 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.578013    2586 scope.go:117] "RemoveContainer" containerID="b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2"
Jul 29 01:30:21.587786 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.587769    2586 scope.go:117] "RemoveContainer" containerID="b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2"
Jul 29 01:30:21.588024 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:30:21.588004    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2\": container with ID starting with b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2 not found: ID does not exist" containerID="b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2"
Jul 29 01:30:21.588100 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.588035    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2"} err="failed to get container status \"b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2\": rpc error: code = NotFound desc = could not find container \"b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2\": container with ID starting with b58aed27a117531c5cb9ddc62f4eda3fd27953e79c4a150758a502046857adb2 not found: ID does not exist"
Jul 29 01:30:21.614319 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.614288    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"]
Jul 29 01:30:21.628013 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.627989    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c9ee4-7686ffdfc7-l28gn"]
Jul 29 01:30:21.664353 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:21.664325    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:22.222486 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:22.222452    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4859dbf2-9100-4493-b672-3e281fac1eb2" path="/var/lib/kubelet/pods/4859dbf2-9100-4493-b672-3e281fac1eb2/volumes"
Jul 29 01:30:22.222862 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:22.222822    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e38ba91-4d39-45c1-9c2b-fe64eae278ae" path="/var/lib/kubelet/pods/9e38ba91-4d39-45c1-9c2b-fe64eae278ae/volumes"
Jul 29 01:30:27.406316 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:27.406284    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:27.507519 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:27.507486    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:27.697929 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:27.697846    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:27.944743 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:27.944705    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:38.454186 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.454135    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"]
Jul 29 01:30:38.454831 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.454547    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v" podUID="219b1bdb-1798-4a4f-9718-8c15291fc742" containerName="maas-api" containerID="cri-o://811251b429b22c5f281870ad1c1ee6a5e6886f578531befbf4dd39bcd1c9a8fc" gracePeriod=30
Jul 29 01:30:38.487984 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.487944    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9"]
Jul 29 01:30:38.489247 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.489191    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:38.492466 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.492436    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ebcf3a-29754810-mnms9"]
Jul 29 01:30:38.517829 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.517796    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"]
Jul 29 01:30:38.518036 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.518001    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l" podUID="b6a7bad0-3e01-469f-85af-3b104ab576a1" containerName="payload-processing" containerID="cri-o://367c530613afc2c27a4b0371eaf5f79aa1b6c8ef4194d715f6c31b326f5d5379" gracePeriod=30
Jul 29 01:30:38.526095 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.525922    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"]
Jul 29 01:30:38.526283 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.526225    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9" podUID="230230a8-349a-4d99-94f9-428fef25f0a2" containerName="payload-pre-processing" containerID="cri-o://12db715f021fdaf62ed967e27ad7114452b71d9ca25fed7543b6e7700799a20b" gracePeriod=30
Jul 29 01:30:38.561407 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.561353    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:38.618948 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.618916    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:38.682403 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.682157    2586 generic.go:356] "Generic (PLEG): container finished" podID="219b1bdb-1798-4a4f-9718-8c15291fc742" containerID="811251b429b22c5f281870ad1c1ee6a5e6886f578531befbf4dd39bcd1c9a8fc" exitCode=0
Jul 29 01:30:38.683560 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.683501    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v" event={"ID":"219b1bdb-1798-4a4f-9718-8c15291fc742","Type":"ContainerDied","Data":"811251b429b22c5f281870ad1c1ee6a5e6886f578531befbf4dd39bcd1c9a8fc"}
Jul 29 01:30:38.685978 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.685950    2586 generic.go:356] "Generic (PLEG): container finished" podID="b6a7bad0-3e01-469f-85af-3b104ab576a1" containerID="367c530613afc2c27a4b0371eaf5f79aa1b6c8ef4194d715f6c31b326f5d5379" exitCode=0
Jul 29 01:30:38.686104 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.686052    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l" event={"ID":"b6a7bad0-3e01-469f-85af-3b104ab576a1","Type":"ContainerDied","Data":"367c530613afc2c27a4b0371eaf5f79aa1b6c8ef4194d715f6c31b326f5d5379"}
Jul 29 01:30:38.689698 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.689664    2586 generic.go:356] "Generic (PLEG): container finished" podID="230230a8-349a-4d99-94f9-428fef25f0a2" containerID="12db715f021fdaf62ed967e27ad7114452b71d9ca25fed7543b6e7700799a20b" exitCode=0
Jul 29 01:30:38.689869 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.689847    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9" event={"ID":"230230a8-349a-4d99-94f9-428fef25f0a2","Type":"ContainerDied","Data":"12db715f021fdaf62ed967e27ad7114452b71d9ca25fed7543b6e7700799a20b"}
Jul 29 01:30:38.719668 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.719637    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:38.737661 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.737639    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:30:38.813553 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.813527    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:30:38.821369 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.821347    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:30:38.866881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.866844    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt7m6\" (UniqueName: \"kubernetes.io/projected/b6a7bad0-3e01-469f-85af-3b104ab576a1-kube-api-access-gt7m6\") pod \"b6a7bad0-3e01-469f-85af-3b104ab576a1\" (UID: \"b6a7bad0-3e01-469f-85af-3b104ab576a1\") "
Jul 29 01:30:38.866881 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.866883    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219b1bdb-1798-4a4f-9718-8c15291fc742-openshift-service-ca\") pod \"219b1bdb-1798-4a4f-9718-8c15291fc742\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") "
Jul 29 01:30:38.867094 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.867029    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a7bad0-3e01-469f-85af-3b104ab576a1-plugins-config-volume\") pod \"b6a7bad0-3e01-469f-85af-3b104ab576a1\" (UID: \"b6a7bad0-3e01-469f-85af-3b104ab576a1\") "
Jul 29 01:30:38.867094 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.867064    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230230a8-349a-4d99-94f9-428fef25f0a2-plugins-config-volume\") pod \"230230a8-349a-4d99-94f9-428fef25f0a2\" (UID: \"230230a8-349a-4d99-94f9-428fef25f0a2\") "
Jul 29 01:30:38.867592 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.867093    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219b1bdb-1798-4a4f-9718-8c15291fc742-maas-api-tls\") pod \"219b1bdb-1798-4a4f-9718-8c15291fc742\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") "
Jul 29 01:30:38.867592 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.867142    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z7hx\" (UniqueName: \"kubernetes.io/projected/230230a8-349a-4d99-94f9-428fef25f0a2-kube-api-access-8z7hx\") pod \"230230a8-349a-4d99-94f9-428fef25f0a2\" (UID: \"230230a8-349a-4d99-94f9-428fef25f0a2\") "
Jul 29 01:30:38.867592 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.867185    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65fz4\" (UniqueName: \"kubernetes.io/projected/219b1bdb-1798-4a4f-9718-8c15291fc742-kube-api-access-65fz4\") pod \"219b1bdb-1798-4a4f-9718-8c15291fc742\" (UID: \"219b1bdb-1798-4a4f-9718-8c15291fc742\") "
Jul 29 01:30:38.867592 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.867383    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219b1bdb-1798-4a4f-9718-8c15291fc742-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "219b1bdb-1798-4a4f-9718-8c15291fc742" (UID: "219b1bdb-1798-4a4f-9718-8c15291fc742"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:30:38.867592 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.867455    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a7bad0-3e01-469f-85af-3b104ab576a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6a7bad0-3e01-469f-85af-3b104ab576a1" (UID: "b6a7bad0-3e01-469f-85af-3b104ab576a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:30:38.867592 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.867512    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219b1bdb-1798-4a4f-9718-8c15291fc742-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:38.867592 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.867504    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230230a8-349a-4d99-94f9-428fef25f0a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "230230a8-349a-4d99-94f9-428fef25f0a2" (UID: "230230a8-349a-4d99-94f9-428fef25f0a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:30:38.869449 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.869426    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219b1bdb-1798-4a4f-9718-8c15291fc742-kube-api-access-65fz4" (OuterVolumeSpecName: "kube-api-access-65fz4") pod "219b1bdb-1798-4a4f-9718-8c15291fc742" (UID: "219b1bdb-1798-4a4f-9718-8c15291fc742"). InnerVolumeSpecName "kube-api-access-65fz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:30:38.869596 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.869581    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230230a8-349a-4d99-94f9-428fef25f0a2-kube-api-access-8z7hx" (OuterVolumeSpecName: "kube-api-access-8z7hx") pod "230230a8-349a-4d99-94f9-428fef25f0a2" (UID: "230230a8-349a-4d99-94f9-428fef25f0a2"). InnerVolumeSpecName "kube-api-access-8z7hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:30:38.869666 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.869605    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a7bad0-3e01-469f-85af-3b104ab576a1-kube-api-access-gt7m6" (OuterVolumeSpecName: "kube-api-access-gt7m6") pod "b6a7bad0-3e01-469f-85af-3b104ab576a1" (UID: "b6a7bad0-3e01-469f-85af-3b104ab576a1"). InnerVolumeSpecName "kube-api-access-gt7m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:30:38.869722 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.869695    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219b1bdb-1798-4a4f-9718-8c15291fc742-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "219b1bdb-1798-4a4f-9718-8c15291fc742" (UID: "219b1bdb-1798-4a4f-9718-8c15291fc742"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:30:38.968711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.968606    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a7bad0-3e01-469f-85af-3b104ab576a1-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:38.968711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.968637    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230230a8-349a-4d99-94f9-428fef25f0a2-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:38.968711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.968651    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219b1bdb-1798-4a4f-9718-8c15291fc742-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:38.968711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.968662    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z7hx\" (UniqueName: \"kubernetes.io/projected/230230a8-349a-4d99-94f9-428fef25f0a2-kube-api-access-8z7hx\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:38.968711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.968671    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65fz4\" (UniqueName: \"kubernetes.io/projected/219b1bdb-1798-4a4f-9718-8c15291fc742-kube-api-access-65fz4\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:38.968711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:38.968680    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7m6\" (UniqueName: \"kubernetes.io/projected/b6a7bad0-3e01-469f-85af-3b104ab576a1-kube-api-access-gt7m6\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:39.022457 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.022425    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:39.696889 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.696857    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"
Jul 29 01:30:39.697385 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.696891    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v" event={"ID":"219b1bdb-1798-4a4f-9718-8c15291fc742","Type":"ContainerDied","Data":"70dc7e56732735ced647b7cd6f0fde2100a29619c4b0aececc8aa839f64a2b35"}
Jul 29 01:30:39.697385 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.696964    2586 scope.go:117] "RemoveContainer" containerID="811251b429b22c5f281870ad1c1ee6a5e6886f578531befbf4dd39bcd1c9a8fc"
Jul 29 01:30:39.698461 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.698448    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"
Jul 29 01:30:39.698533 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.698460    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l" event={"ID":"b6a7bad0-3e01-469f-85af-3b104ab576a1","Type":"ContainerDied","Data":"32d70f6b1be13f5d28c29663d88623672b26dca39665ce7074c6cf7fa521be73"}
Jul 29 01:30:39.700007 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.699976    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9" event={"ID":"230230a8-349a-4d99-94f9-428fef25f0a2","Type":"ContainerDied","Data":"c2922b051eb1645930ee6e536df212eb01de0002f8122b2f51a83614dea59244"}
Jul 29 01:30:39.700007 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.700000    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"
Jul 29 01:30:39.707388 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.707370    2586 scope.go:117] "RemoveContainer" containerID="367c530613afc2c27a4b0371eaf5f79aa1b6c8ef4194d715f6c31b326f5d5379"
Jul 29 01:30:39.716116 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.716100    2586 scope.go:117] "RemoveContainer" containerID="12db715f021fdaf62ed967e27ad7114452b71d9ca25fed7543b6e7700799a20b"
Jul 29 01:30:39.726078 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.726055    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"]
Jul 29 01:30:39.728622 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.728601    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebcf3a-7686b7d8f7-trd9v"]
Jul 29 01:30:39.739932 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.739902    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"]
Jul 29 01:30:39.746215 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.746171    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ebcf3a-79f6bd4c4-lwk8l"]
Jul 29 01:30:39.758655 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.758601    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"]
Jul 29 01:30:39.763226 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:39.763185    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ebcf3a-584549464-rgkp9"]
Jul 29 01:30:40.222033 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:40.221999    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d8537d4-3247-488c-8234-7b79c358055d" path="/var/lib/kubelet/pods/0d8537d4-3247-488c-8234-7b79c358055d/volumes"
Jul 29 01:30:40.222484 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:40.222463    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="219b1bdb-1798-4a4f-9718-8c15291fc742" path="/var/lib/kubelet/pods/219b1bdb-1798-4a4f-9718-8c15291fc742/volumes"
Jul 29 01:30:40.222859 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:40.222842    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230230a8-349a-4d99-94f9-428fef25f0a2" path="/var/lib/kubelet/pods/230230a8-349a-4d99-94f9-428fef25f0a2/volumes"
Jul 29 01:30:40.223229 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:40.223209    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a7bad0-3e01-469f-85af-3b104ab576a1" path="/var/lib/kubelet/pods/b6a7bad0-3e01-469f-85af-3b104ab576a1/volumes"
Jul 29 01:30:44.844434 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:44.844391    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:44.944603 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:44.944566    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:45.102440 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.102342    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:45.342092 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.342057    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:45.457413 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.457318    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"]
Jul 29 01:30:45.457936 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.457916    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a7bad0-3e01-469f-85af-3b104ab576a1" containerName="payload-processing"
Jul 29 01:30:45.457985 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.457940    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a7bad0-3e01-469f-85af-3b104ab576a1" containerName="payload-processing"
Jul 29 01:30:45.457985 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.457968    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219b1bdb-1798-4a4f-9718-8c15291fc742" containerName="maas-api"
Jul 29 01:30:45.457985 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.457976    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="219b1bdb-1798-4a4f-9718-8c15291fc742" containerName="maas-api"
Jul 29 01:30:45.458081 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.457994    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4859dbf2-9100-4493-b672-3e281fac1eb2" containerName="revoke-keys"
Jul 29 01:30:45.458081 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458004    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4859dbf2-9100-4493-b672-3e281fac1eb2" containerName="revoke-keys"
Jul 29 01:30:45.458081 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458018    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e38ba91-4d39-45c1-9c2b-fe64eae278ae" containerName="maas-api"
Jul 29 01:30:45.458081 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458028    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e38ba91-4d39-45c1-9c2b-fe64eae278ae" containerName="maas-api"
Jul 29 01:30:45.458081 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458040    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230230a8-349a-4d99-94f9-428fef25f0a2" containerName="payload-pre-processing"
Jul 29 01:30:45.458081 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458049    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="230230a8-349a-4d99-94f9-428fef25f0a2" containerName="payload-pre-processing"
Jul 29 01:30:45.458280 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458137    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e38ba91-4d39-45c1-9c2b-fe64eae278ae" containerName="maas-api"
Jul 29 01:30:45.458280 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458149    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="219b1bdb-1798-4a4f-9718-8c15291fc742" containerName="maas-api"
Jul 29 01:30:45.458280 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458160    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a7bad0-3e01-469f-85af-3b104ab576a1" containerName="payload-processing"
Jul 29 01:30:45.458280 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458169    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="230230a8-349a-4d99-94f9-428fef25f0a2" containerName="payload-pre-processing"
Jul 29 01:30:45.458280 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.458182    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4859dbf2-9100-4493-b672-3e281fac1eb2" containerName="revoke-keys"
Jul 29 01:30:45.463382 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.463352    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"
Jul 29 01:30:45.466693 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.466662    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4nzwm\""
Jul 29 01:30:45.467767 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.467736    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"]
Jul 29 01:30:45.531849 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.531801    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c\" (UID: \"fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"
Jul 29 01:30:45.632592 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.632547    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c\" (UID: \"fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"
Jul 29 01:30:45.633217 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.633170    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c\" (UID: \"fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"
Jul 29 01:30:45.778223 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.778082    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"
Jul 29 01:30:45.921951 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:45.921925    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"]
Jul 29 01:30:45.923975 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:30:45.923943    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc39b7c6_5ce1_4f8c_bdf1_cbb8bef2a1e9.slice/crio-63d4488d07ebaf3e50686f1759a5562d85cca047b84912a31a906d5de83b93e4 WatchSource:0}: Error finding container 63d4488d07ebaf3e50686f1759a5562d85cca047b84912a31a906d5de83b93e4: Status 404 returned error can't find the container with id 63d4488d07ebaf3e50686f1759a5562d85cca047b84912a31a906d5de83b93e4
Jul 29 01:30:46.745325 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:46.745288    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c" event={"ID":"fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9","Type":"ContainerStarted","Data":"ce3aa7bdc2b833aec2f4cd1900cbcdd5c018e5d6bc8585df15cdca8f306fc7d5"}
Jul 29 01:30:46.745542 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:46.745364    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c" event={"ID":"fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9","Type":"ContainerStarted","Data":"63d4488d07ebaf3e50686f1759a5562d85cca047b84912a31a906d5de83b93e4"}
Jul 29 01:30:46.761492 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:46.761441    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c" podStartSLOduration=1.7614247779999999 podStartE2EDuration="1.761424778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:30:46.760979509 +0000 UTC m=+3277.069907352" watchObservedRunningTime="2026-07-29 01:30:46.761424778 +0000 UTC m=+3277.070352620"
Jul 29 01:30:47.752593 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:47.752557    2586 generic.go:356] "Generic (PLEG): container finished" podID="fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9" containerID="ce3aa7bdc2b833aec2f4cd1900cbcdd5c018e5d6bc8585df15cdca8f306fc7d5" exitCode=0
Jul 29 01:30:47.752971 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:47.752642    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c" event={"ID":"fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9","Type":"ContainerDied","Data":"ce3aa7bdc2b833aec2f4cd1900cbcdd5c018e5d6bc8585df15cdca8f306fc7d5"}
Jul 29 01:30:48.902931 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:48.902899    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"
Jul 29 01:30:48.961785 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:48.961744    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9-maas-api-service-ca\") pod \"fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9\" (UID: \"fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9\") "
Jul 29 01:30:48.962115 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:48.962089    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9" (UID: "fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:30:49.062548 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:49.062518    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9-maas-api-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:49.765564 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:49.765533    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"
Jul 29 01:30:49.765760 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:49.765567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c" event={"ID":"fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9","Type":"ContainerDied","Data":"63d4488d07ebaf3e50686f1759a5562d85cca047b84912a31a906d5de83b93e4"}
Jul 29 01:30:49.765760 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:49.765616    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63d4488d07ebaf3e50686f1759a5562d85cca047b84912a31a906d5de83b93e4"
Jul 29 01:30:53.003926 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:53.003894    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:53.104952 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:53.104917    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:56.034106 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.034063    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"]
Jul 29 01:30:56.034619 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.034352    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf" podUID="25cc489d-dcc6-42f9-8a75-02d55af2bdfc" containerName="maas-api" containerID="cri-o://7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329" gracePeriod=30
Jul 29 01:30:56.057452 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.057419    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:56.062770 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.062740    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm"]
Jul 29 01:30:56.069226 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.068489    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-afd6d5-29754810-2hgdm"]
Jul 29 01:30:56.098168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.098134    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"]
Jul 29 01:30:56.098470 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.098442    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2" podUID="dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf" containerName="payload-processing" containerID="cri-o://04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11" gracePeriod=30
Jul 29 01:30:56.105085 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.105051    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"]
Jul 29 01:30:56.105352 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.105302    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8" podUID="29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44" containerName="payload-pre-processing" containerID="cri-o://472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7" gracePeriod=30
Jul 29 01:30:56.157351 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.157319    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:56.223669 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.223626    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef8e095c-3646-42d8-bc69-1e841e2aa0a1" path="/var/lib/kubelet/pods/ef8e095c-3646-42d8-bc69-1e841e2aa0a1/volumes"
Jul 29 01:30:56.231042 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.230992    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"]
Jul 29 01:30:56.233711 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.233665    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:56.235219 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.235176    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-afd6d5-fm47c"]
Jul 29 01:30:56.294784 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.294654    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:56.417883 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.417861    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:30:56.420770 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.420748    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:30:56.434334 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.434309    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:30:56.532887 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.532852    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4mjg\" (UniqueName: \"kubernetes.io/projected/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-kube-api-access-m4mjg\") pod \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") "
Jul 29 01:30:56.533086 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.532916    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-maas-api-tls\") pod \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") "
Jul 29 01:30:56.533086 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.532940    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5x95\" (UniqueName: \"kubernetes.io/projected/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-kube-api-access-f5x95\") pod \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\" (UID: \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\") "
Jul 29 01:30:56.533086 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.532961    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcrd4\" (UniqueName: \"kubernetes.io/projected/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-kube-api-access-rcrd4\") pod \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\" (UID: \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\") "
Jul 29 01:30:56.533086 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.532981    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-openshift-service-ca\") pod \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\" (UID: \"25cc489d-dcc6-42f9-8a75-02d55af2bdfc\") "
Jul 29 01:30:56.533086 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.533005    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-plugins-config-volume\") pod \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\" (UID: \"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf\") "
Jul 29 01:30:56.533086 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.533037    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-plugins-config-volume\") pod \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\" (UID: \"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44\") "
Jul 29 01:30:56.533477 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.533445    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf" (UID: "dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:30:56.533556 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.533525    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25cc489d-dcc6-42f9-8a75-02d55af2bdfc" (UID: "25cc489d-dcc6-42f9-8a75-02d55af2bdfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:30:56.533618 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.533558    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44" (UID: "29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:30:56.535331 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.535300    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-kube-api-access-rcrd4" (OuterVolumeSpecName: "kube-api-access-rcrd4") pod "dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf" (UID: "dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf"). InnerVolumeSpecName "kube-api-access-rcrd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:30:56.535426 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.535334    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25cc489d-dcc6-42f9-8a75-02d55af2bdfc" (UID: "25cc489d-dcc6-42f9-8a75-02d55af2bdfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:30:56.535426 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.535355    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-kube-api-access-m4mjg" (OuterVolumeSpecName: "kube-api-access-m4mjg") pod "25cc489d-dcc6-42f9-8a75-02d55af2bdfc" (UID: "25cc489d-dcc6-42f9-8a75-02d55af2bdfc"). InnerVolumeSpecName "kube-api-access-m4mjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:30:56.535426 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.535362    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-kube-api-access-f5x95" (OuterVolumeSpecName: "kube-api-access-f5x95") pod "29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44" (UID: "29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44"). InnerVolumeSpecName "kube-api-access-f5x95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:30:56.602103 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.602070    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:30:56.633878 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.633841    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4mjg\" (UniqueName: \"kubernetes.io/projected/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-kube-api-access-m4mjg\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:56.633878 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.633875    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-maas-api-tls\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:56.633878 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.633885    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5x95\" (UniqueName: \"kubernetes.io/projected/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-kube-api-access-f5x95\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:56.634098 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.633897    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcrd4\" (UniqueName: \"kubernetes.io/projected/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-kube-api-access-rcrd4\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:56.634098 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.633933    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc489d-dcc6-42f9-8a75-02d55af2bdfc-openshift-service-ca\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:56.634098 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.633942    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:56.634098 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.633951    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44-plugins-config-volume\") on node \"ip-10-0-132-227.ec2.internal\" DevicePath \"\""
Jul 29 01:30:56.807648 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.807611    2586 generic.go:356] "Generic (PLEG): container finished" podID="29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44" containerID="472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7" exitCode=0
Jul 29 01:30:56.807648 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.807648    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"
Jul 29 01:30:56.807889 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.807685    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8" event={"ID":"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44","Type":"ContainerDied","Data":"472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7"}
Jul 29 01:30:56.807889 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.807735    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8" event={"ID":"29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44","Type":"ContainerDied","Data":"2563d27c7aed1534c5f554fff31d8328c7b7cba22066cc7cdca44a169aa8119c"}
Jul 29 01:30:56.807889 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.807761    2586 scope.go:117] "RemoveContainer" containerID="472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7"
Jul 29 01:30:56.809168 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.809146    2586 generic.go:356] "Generic (PLEG): container finished" podID="dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf" containerID="04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11" exitCode=0
Jul 29 01:30:56.809293 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.809191    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"
Jul 29 01:30:56.809293 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.809227    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2" event={"ID":"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf","Type":"ContainerDied","Data":"04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11"}
Jul 29 01:30:56.809409 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.809313    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2" event={"ID":"dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf","Type":"ContainerDied","Data":"964d0ad12eda7dee77a206a740139edc600d11819a6ec7e6bc0d9d9992d4ea40"}
Jul 29 01:30:56.810875 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.810847    2586 generic.go:356] "Generic (PLEG): container finished" podID="25cc489d-dcc6-42f9-8a75-02d55af2bdfc" containerID="7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329" exitCode=0
Jul 29 01:30:56.811002 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.810894    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf" event={"ID":"25cc489d-dcc6-42f9-8a75-02d55af2bdfc","Type":"ContainerDied","Data":"7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329"}
Jul 29 01:30:56.811002 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.810900    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"
Jul 29 01:30:56.811002 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.810934    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf" event={"ID":"25cc489d-dcc6-42f9-8a75-02d55af2bdfc","Type":"ContainerDied","Data":"55d35f8e4b59708f5957ae1856de0845d5e1faafb51bdb8fea5a10e8fea6651d"}
Jul 29 01:30:56.818214 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.818177    2586 scope.go:117] "RemoveContainer" containerID="472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7"
Jul 29 01:30:56.818482 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:30:56.818463    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7\": container with ID starting with 472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7 not found: ID does not exist" containerID="472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7"
Jul 29 01:30:56.818576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.818493    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7"} err="failed to get container status \"472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7\": rpc error: code = NotFound desc = could not find container \"472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7\": container with ID starting with 472088618c8af9ad28beb78f161f3f4a8871f32d1b89923fe10826f5a61f95a7 not found: ID does not exist"
Jul 29 01:30:56.818576 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.818516    2586 scope.go:117] "RemoveContainer" containerID="04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11"
Jul 29 01:30:56.827165 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.827146    2586 scope.go:117] "RemoveContainer" containerID="04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11"
Jul 29 01:30:56.827439 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:30:56.827422    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11\": container with ID starting with 04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11 not found: ID does not exist" containerID="04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11"
Jul 29 01:30:56.827507 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.827446    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11"} err="failed to get container status \"04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11\": rpc error: code = NotFound desc = could not find container \"04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11\": container with ID starting with 04fadf748277f2f4aa43775aa39a099eace4d04a7d4b7751cecd05abd5e93f11 not found: ID does not exist"
Jul 29 01:30:56.827507 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.827462    2586 scope.go:117] "RemoveContainer" containerID="7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329"
Jul 29 01:30:56.832897 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.832874    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"]
Jul 29 01:30:56.837963 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.837936    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-afd6d5-7b97ccf5fb-lctm8"]
Jul 29 01:30:56.841530 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.841509    2586 scope.go:117] "RemoveContainer" containerID="7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329"
Jul 29 01:30:56.841824 ip-10-0-132-227 kubenswrapper[2586]: E0729 01:30:56.841806    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329\": container with ID starting with 7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329 not found: ID does not exist" containerID="7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329"
Jul 29 01:30:56.841873 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.841836    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329"} err="failed to get container status \"7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329\": rpc error: code = NotFound desc = could not find container \"7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329\": container with ID starting with 7d238f9da2d2518cdd56f7472e54808fd11e428b13eab3f91ce01a81bd6d1329 not found: ID does not exist"
Jul 29 01:30:56.851014 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.850988    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"]
Jul 29 01:30:56.853372 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.853316    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-afd6d5-5957c4cbf8-t2ddf"]
Jul 29 01:30:56.863640 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.863614    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"]
Jul 29 01:30:56.867742 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:56.867714    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-afd6d5-6d7b7659d6-r25d2"]
Jul 29 01:30:58.222892 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:58.222857    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25cc489d-dcc6-42f9-8a75-02d55af2bdfc" path="/var/lib/kubelet/pods/25cc489d-dcc6-42f9-8a75-02d55af2bdfc/volumes"
Jul 29 01:30:58.223319 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:58.223228    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44" path="/var/lib/kubelet/pods/29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44/volumes"
Jul 29 01:30:58.223545 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:58.223532    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf" path="/var/lib/kubelet/pods/dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf/volumes"
Jul 29 01:30:58.223812 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:30:58.223800    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9" path="/var/lib/kubelet/pods/fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9/volumes"
Jul 29 01:31:02.572491 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:02.572451    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:31:02.674054 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:02.674018    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:31:02.833847 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:02.833745    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:31:03.059749 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:03.059700    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:31:09.108539 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:09.108505    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787f676965-kjbrt_d222f951-af1d-4bff-8c93-5a7225cdba9a/manager/0.log"
Jul 29 01:31:09.238777 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:09.238750    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zrpn4_9b02ca09-a952-49e2-8df8-40f0d7c716f6/server/0.log"
Jul 29 01:31:09.375988 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:09.375897    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jbxvv_5f2bd4e2-4712-4840-800f-eb424f728dc9/manager/0.log"
Jul 29 01:31:09.484410 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:09.484377    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf6bc8b6f-88px6_603ac805-e47a-454e-a1bf-8d761fbcf4f9/copy-manifests/0.log"
Jul 29 01:31:09.496355 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:09.496317    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf6bc8b6f-88px6_603ac805-e47a-454e-a1bf-8d761fbcf4f9/manager/0.log"
Jul 29 01:31:10.227100 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:10.227071    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754780-pjppn_131ddd08-2d3b-4a59-be83-acc69e45df2a/cleanup/0.log"
Jul 29 01:31:10.346433 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:10.346401    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754795-9x4vv_78805b92-92b6-4255-938a-7a6228005a83/cleanup/0.log"
Jul 29 01:31:11.356744 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.356709    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm_e6188400-01f0-4993-baef-a71a9a5dfdee/util/0.log"
Jul 29 01:31:11.362321 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.362297    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm_e6188400-01f0-4993-baef-a71a9a5dfdee/pull/0.log"
Jul 29 01:31:11.367433 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.367411    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm_e6188400-01f0-4993-baef-a71a9a5dfdee/extract/0.log"
Jul 29 01:31:11.479023 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.478988    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6_27ecacce-22bf-4746-923c-20d2bd2c8e81/extract/0.log"
Jul 29 01:31:11.485261 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.485240    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6_27ecacce-22bf-4746-923c-20d2bd2c8e81/util/0.log"
Jul 29 01:31:11.490804 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.490773    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6_27ecacce-22bf-4746-923c-20d2bd2c8e81/pull/0.log"
Jul 29 01:31:11.612217 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.612124    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m_5ba67441-b9a7-439f-ad86-c3ffecf6af5c/extract/0.log"
Jul 29 01:31:11.619752 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.619729    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m_5ba67441-b9a7-439f-ad86-c3ffecf6af5c/util/0.log"
Jul 29 01:31:11.632008 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.631981    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m_5ba67441-b9a7-439f-ad86-c3ffecf6af5c/pull/0.log"
Jul 29 01:31:11.754285 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.754256    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf_96281e7d-5ebb-4387-816c-54bf9cf67b0e/pull/0.log"
Jul 29 01:31:11.767037 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.767010    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf_96281e7d-5ebb-4387-816c-54bf9cf67b0e/extract/0.log"
Jul 29 01:31:11.778861 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:11.778833    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf_96281e7d-5ebb-4387-816c-54bf9cf67b0e/util/0.log"
Jul 29 01:31:12.035452 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:12.035350    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bfpgh_cc62d8dc-1eea-4ef7-b61a-429f104447ae/manager/0.log"
Jul 29 01:31:12.158120 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:12.153021    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7gdvt_9071ce25-d8c5-4666-9b8c-8b528dd5c056/manager/0.log"
Jul 29 01:31:12.384757 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:12.384724    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5qwr9_82308f1b-a52a-4f42-ba29-8d3dbbdd454e/registry-server/0.log"
Jul 29 01:31:12.511415 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:12.511378    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ggbbl_386d3ba9-c326-4224-9c08-d3104525dc0a/manager/0.log"
Jul 29 01:31:12.742862 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:12.742775    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lm7xw_ee7b1eaf-3df1-49ba-b8af-522596a7e862/manager/0.log"
Jul 29 01:31:13.227753 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:13.227721    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tpssf_86d7d553-2b8c-46cf-9884-840447175377/discovery/0.log"
Jul 29 01:31:13.449058 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:13.449031    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75d55f5564-tbqst_5373cb16-6de6-4fc9-a6b8-a7b7207dc945/kube-auth-proxy/0.log"
Jul 29 01:31:13.797538 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:13.797506    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5cvtz_5a369ca0-d428-4bee-bc25-23a4ea6dd7fe/payload-processing/0.log"
Jul 29 01:31:14.246886 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:14.246848    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-25zfm_04626e14-481a-414c-97a0-6e0d3de7a19e/main/0.log"
Jul 29 01:31:14.474017 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:14.473986    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-46k88_722e2b17-ec12-4396-8303-23c324aaff37/main/0.log"
Jul 29 01:31:20.294161 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294125    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-99rkt/must-gather-v6l7w"]
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294513    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25cc489d-dcc6-42f9-8a75-02d55af2bdfc" containerName="maas-api"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294524    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="25cc489d-dcc6-42f9-8a75-02d55af2bdfc" containerName="maas-api"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294537    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf" containerName="payload-processing"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294543    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf" containerName="payload-processing"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294550    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9" containerName="revoke-keys"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294556    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9" containerName="revoke-keys"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294570    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44" containerName="payload-pre-processing"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294575    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44" containerName="payload-pre-processing"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294630    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="29163a3c-d1ff-4f8f-b311-9ec5b0e1bd44" containerName="payload-pre-processing"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294638    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc39b7c6-5ce1-4f8c-bdf1-cbb8bef2a1e9" containerName="revoke-keys"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294645    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="25cc489d-dcc6-42f9-8a75-02d55af2bdfc" containerName="maas-api"
Jul 29 01:31:20.294652 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.294651    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dede6f03-bcc4-4d08-8ca1-eaa2b21aa4bf" containerName="payload-processing"
Jul 29 01:31:20.299743 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.299716    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99rkt/must-gather-v6l7w"
Jul 29 01:31:20.302655 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.302627    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-99rkt\"/\"kube-root-ca.crt\""
Jul 29 01:31:20.302814 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.302703    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-99rkt\"/\"openshift-service-ca.crt\""
Jul 29 01:31:20.303950 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.303931    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-99rkt\"/\"default-dockercfg-fls7h\""
Jul 29 01:31:20.309358 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.309329    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-99rkt/must-gather-v6l7w"]
Jul 29 01:31:20.311367 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.311337    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:31:20.349732 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.349697    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8vlm\" (UniqueName: \"kubernetes.io/projected/930eca1c-8eed-418c-80cf-a80f521d15b7-kube-api-access-l8vlm\") pod \"must-gather-v6l7w\" (UID: \"930eca1c-8eed-418c-80cf-a80f521d15b7\") " pod="openshift-must-gather-99rkt/must-gather-v6l7w"
Jul 29 01:31:20.349732 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.349736    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/930eca1c-8eed-418c-80cf-a80f521d15b7-must-gather-output\") pod \"must-gather-v6l7w\" (UID: \"930eca1c-8eed-418c-80cf-a80f521d15b7\") " pod="openshift-must-gather-99rkt/must-gather-v6l7w"
Jul 29 01:31:20.388257 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.388220    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log" limit=1024
Jul 29 01:31:20.450568 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.450526    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8vlm\" (UniqueName: \"kubernetes.io/projected/930eca1c-8eed-418c-80cf-a80f521d15b7-kube-api-access-l8vlm\") pod \"must-gather-v6l7w\" (UID: \"930eca1c-8eed-418c-80cf-a80f521d15b7\") " pod="openshift-must-gather-99rkt/must-gather-v6l7w"
Jul 29 01:31:20.450742 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.450573    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/930eca1c-8eed-418c-80cf-a80f521d15b7-must-gather-output\") pod \"must-gather-v6l7w\" (UID: \"930eca1c-8eed-418c-80cf-a80f521d15b7\") " pod="openshift-must-gather-99rkt/must-gather-v6l7w"
Jul 29 01:31:20.450971 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.450950    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/930eca1c-8eed-418c-80cf-a80f521d15b7-must-gather-output\") pod \"must-gather-v6l7w\" (UID: \"930eca1c-8eed-418c-80cf-a80f521d15b7\") " pod="openshift-must-gather-99rkt/must-gather-v6l7w"
Jul 29 01:31:20.460071 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.460038    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8vlm\" (UniqueName: \"kubernetes.io/projected/930eca1c-8eed-418c-80cf-a80f521d15b7-kube-api-access-l8vlm\") pod \"must-gather-v6l7w\" (UID: \"930eca1c-8eed-418c-80cf-a80f521d15b7\") " pod="openshift-must-gather-99rkt/must-gather-v6l7w"
Jul 29 01:31:20.609869 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.609821    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99rkt/must-gather-v6l7w"
Jul 29 01:31:20.756797 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.756770    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-99rkt/must-gather-v6l7w"]
Jul 29 01:31:20.758842 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:31:20.758805    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod930eca1c_8eed_418c_80cf_a80f521d15b7.slice/crio-63f819cbc9872e3a4fab490a0b36797927ba1a2cdf875ef578f8234938e66a36 WatchSource:0}: Error finding container 63f819cbc9872e3a4fab490a0b36797927ba1a2cdf875ef578f8234938e66a36: Status 404 returned error can't find the container with id 63f819cbc9872e3a4fab490a0b36797927ba1a2cdf875ef578f8234938e66a36
Jul 29 01:31:20.761255 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.761233    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:31:20.960987 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:20.960891    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99rkt/must-gather-v6l7w" event={"ID":"930eca1c-8eed-418c-80cf-a80f521d15b7","Type":"ContainerStarted","Data":"63f819cbc9872e3a4fab490a0b36797927ba1a2cdf875ef578f8234938e66a36"}
Jul 29 01:31:21.970874 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:21.970766    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99rkt/must-gather-v6l7w" event={"ID":"930eca1c-8eed-418c-80cf-a80f521d15b7","Type":"ContainerStarted","Data":"de8ae3a4aca874c12a6c4e9b54e13d7938b47c1f2fe43b088589a2352b8cde71"}
Jul 29 01:31:21.970874 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:21.970820    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99rkt/must-gather-v6l7w" event={"ID":"930eca1c-8eed-418c-80cf-a80f521d15b7","Type":"ContainerStarted","Data":"d86a6fe8a3f956dbfe06e27804e646bb162485aa75e166047a401c10ed48ccb8"}
Jul 29 01:31:21.988706 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:21.988633    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-99rkt/must-gather-v6l7w" podStartSLOduration=1.165282682 podStartE2EDuration="1.988608167s" totalImagesPullingTime="823.325485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:31:21.98637011 +0000 UTC m=+3312.295297954" watchObservedRunningTime="2026-07-29 01:31:21.988608167 +0000 UTC m=+3312.297536014"
Jul 29 01:31:23.128813 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:23.128777    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrq9b_5c14500d-ea5d-4433-ac3c-2df82c0ef982/global-pull-secret-syncer/0.log"
Jul 29 01:31:23.261153 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:23.261106    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2t4gv_acbd27bf-81aa-49e9-9c59-d58bcf765cd8/konnectivity-agent/0.log"
Jul 29 01:31:23.350213 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:23.350176    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-227.ec2.internal_8f053b035387553a0f709b762fddad9a/haproxy/0.log"
Jul 29 01:31:27.910910 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:27.910879    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm_e6188400-01f0-4993-baef-a71a9a5dfdee/extract/0.log"
Jul 29 01:31:27.944858 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:27.944758    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm_e6188400-01f0-4993-baef-a71a9a5dfdee/util/0.log"
Jul 29 01:31:27.965872 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:27.965775    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cr2cm_e6188400-01f0-4993-baef-a71a9a5dfdee/pull/0.log"
Jul 29 01:31:28.014665 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.014633    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6_27ecacce-22bf-4746-923c-20d2bd2c8e81/extract/0.log"
Jul 29 01:31:28.038690 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.038645    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6_27ecacce-22bf-4746-923c-20d2bd2c8e81/util/0.log"
Jul 29 01:31:28.065973 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.065937    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rsg6_27ecacce-22bf-4746-923c-20d2bd2c8e81/pull/0.log"
Jul 29 01:31:28.093691 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.093546    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m_5ba67441-b9a7-439f-ad86-c3ffecf6af5c/extract/0.log"
Jul 29 01:31:28.121972 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.121801    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m_5ba67441-b9a7-439f-ad86-c3ffecf6af5c/util/0.log"
Jul 29 01:31:28.144623 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.144584    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kbv7m_5ba67441-b9a7-439f-ad86-c3ffecf6af5c/pull/0.log"
Jul 29 01:31:28.171252 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.171156    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf_96281e7d-5ebb-4387-816c-54bf9cf67b0e/extract/0.log"
Jul 29 01:31:28.192241 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.192183    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf_96281e7d-5ebb-4387-816c-54bf9cf67b0e/util/0.log"
Jul 29 01:31:28.225063 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.225032    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gkjgf_96281e7d-5ebb-4387-816c-54bf9cf67b0e/pull/0.log"
Jul 29 01:31:28.595784 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.595747    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bfpgh_cc62d8dc-1eea-4ef7-b61a-429f104447ae/manager/0.log"
Jul 29 01:31:28.626079 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.626041    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7gdvt_9071ce25-d8c5-4666-9b8c-8b528dd5c056/manager/0.log"
Jul 29 01:31:28.701693 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.701660    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5qwr9_82308f1b-a52a-4f42-ba29-8d3dbbdd454e/registry-server/0.log"
Jul 29 01:31:28.895036 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:28.894946    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ggbbl_386d3ba9-c326-4224-9c08-d3104525dc0a/manager/0.log"
Jul 29 01:31:29.003390 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:29.003317    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lm7xw_ee7b1eaf-3df1-49ba-b8af-522596a7e862/manager/0.log"
Jul 29 01:31:30.864558 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:30.864493    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1a879a72-fe00-4a52-b335-29b33ec030c3/alertmanager/0.log"
Jul 29 01:31:30.888254 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:30.888221    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1a879a72-fe00-4a52-b335-29b33ec030c3/config-reloader/0.log"
Jul 29 01:31:30.912800 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:30.912744    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1a879a72-fe00-4a52-b335-29b33ec030c3/kube-rbac-proxy-web/0.log"
Jul 29 01:31:30.935077 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:30.935039    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1a879a72-fe00-4a52-b335-29b33ec030c3/kube-rbac-proxy/0.log"
Jul 29 01:31:30.958181 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:30.958148    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1a879a72-fe00-4a52-b335-29b33ec030c3/kube-rbac-proxy-metric/0.log"
Jul 29 01:31:30.985003 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:30.984973    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1a879a72-fe00-4a52-b335-29b33ec030c3/prom-label-proxy/0.log"
Jul 29 01:31:31.007838 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:31.007813    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1a879a72-fe00-4a52-b335-29b33ec030c3/init-config-reloader/0.log"
Jul 29 01:31:31.179526 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:31.179452    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-cfqqp_bd0a7f79-f912-476a-8779-e1590181fc39/monitoring-plugin/0.log"
Jul 29 01:31:31.283489 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:31.283457    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgqz8_c542210c-ab65-4f1d-8573-37fe1ba9b955/node-exporter/0.log"
Jul 29 01:31:31.306323 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:31.306275    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgqz8_c542210c-ab65-4f1d-8573-37fe1ba9b955/kube-rbac-proxy/0.log"
Jul 29 01:31:31.348037 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:31.347990    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgqz8_c542210c-ab65-4f1d-8573-37fe1ba9b955/init-textfile/0.log"
Jul 29 01:31:31.701570 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:31.701536    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-lp4qj_21b3d742-675b-43c9-b0b5-bf0e2730f013/prometheus-operator/0.log"
Jul 29 01:31:31.718799 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:31.718716    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-lp4qj_21b3d742-675b-43c9-b0b5-bf0e2730f013/kube-rbac-proxy/0.log"
Jul 29 01:31:31.743443 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:31.743366    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-vb4w7_638b20e8-9b2b-4216-a1d0-d60a186ae13c/prometheus-operator-admission-webhook/0.log"
Jul 29 01:31:33.582787 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.582748    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"]
Jul 29 01:31:33.591155 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.591125    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.600035 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.599317    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"]
Jul 29 01:31:33.698735 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.698690    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-proc\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.698941 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.698751    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-sys\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.698941 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.698780    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-lib-modules\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.698941 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.698810    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4h7\" (UniqueName: \"kubernetes.io/projected/d3b70c76-637f-4b9e-8c55-e7acde23f49f-kube-api-access-cp4h7\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.698941 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.698930    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-podres\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.800469 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.800425    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-proc\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.800832 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.800812    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-sys\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.801017 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.801001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-lib-modules\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.801308 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.801289    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4h7\" (UniqueName: \"kubernetes.io/projected/d3b70c76-637f-4b9e-8c55-e7acde23f49f-kube-api-access-cp4h7\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.801878 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.801856    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-podres\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.802208 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.801240    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-lib-modules\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.802334 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.800967    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-sys\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.802425 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.800760    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-proc\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.802510 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.802085    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3b70c76-637f-4b9e-8c55-e7acde23f49f-podres\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.811623 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.811587    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4h7\" (UniqueName: \"kubernetes.io/projected/d3b70c76-637f-4b9e-8c55-e7acde23f49f-kube-api-access-cp4h7\") pod \"perf-node-gather-daemonset-kv6dz\" (UID: \"d3b70c76-637f-4b9e-8c55-e7acde23f49f\") " pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:33.906950 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:33.906852    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:34.088131 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:34.087706    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"]
Jul 29 01:31:34.090816 ip-10-0-132-227 kubenswrapper[2586]: W0729 01:31:34.090762    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd3b70c76_637f_4b9e_8c55_e7acde23f49f.slice/crio-4fc4969e995e0dffe5ceff14c1b373f3144c2a42135963a1df2555765b8b3d65 WatchSource:0}: Error finding container 4fc4969e995e0dffe5ceff14c1b373f3144c2a42135963a1df2555765b8b3d65: Status 404 returned error can't find the container with id 4fc4969e995e0dffe5ceff14c1b373f3144c2a42135963a1df2555765b8b3d65
Jul 29 01:31:34.310544 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:34.310509    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-778ff5dc99-xj9wj_e50c874b-77a2-4a2f-a2e4-9fa281363a19/console/0.log"
Jul 29 01:31:35.086593 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:35.086538    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz" event={"ID":"d3b70c76-637f-4b9e-8c55-e7acde23f49f","Type":"ContainerStarted","Data":"09dca03c1ddef150e92aac9e169473b39f1e4990040d4419d93360b38e7fe387"}
Jul 29 01:31:35.087020 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:35.086621    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:35.087020 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:35.086646    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz" event={"ID":"d3b70c76-637f-4b9e-8c55-e7acde23f49f","Type":"ContainerStarted","Data":"4fc4969e995e0dffe5ceff14c1b373f3144c2a42135963a1df2555765b8b3d65"}
Jul 29 01:31:35.104446 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:35.104381    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz" podStartSLOduration=2.104355718 podStartE2EDuration="2.104355718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:31:35.102972821 +0000 UTC m=+3325.411900665" watchObservedRunningTime="2026-07-29 01:31:35.104355718 +0000 UTC m=+3325.413283561"
Jul 29 01:31:36.093121 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:36.093090    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqrrw_b5e0eb10-33c8-4f4a-9076-f3ccda11708c/dns/0.log"
Jul 29 01:31:36.114910 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:36.114884    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqrrw_b5e0eb10-33c8-4f4a-9076-f3ccda11708c/kube-rbac-proxy/0.log"
Jul 29 01:31:36.162366 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:36.162336    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-krk2j_a115e159-a003-4bdc-8e43-94ac470b5d1a/dns-node-resolver/0.log"
Jul 29 01:31:36.854226 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:36.854175    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-74znh_e7a2ae25-317e-403b-a257-b993fe994cf2/node-ca/0.log"
Jul 29 01:31:37.963982 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:37.963946    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tpssf_86d7d553-2b8c-46cf-9884-840447175377/discovery/0.log"
Jul 29 01:31:38.008958 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:38.008920    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75d55f5564-tbqst_5373cb16-6de6-4fc9-a6b8-a7b7207dc945/kube-auth-proxy/0.log"
Jul 29 01:31:38.122462 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:38.122426    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5cvtz_5a369ca0-d428-4bee-bc25-23a4ea6dd7fe/payload-processing/0.log"
Jul 29 01:31:38.979469 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:38.979431    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5cv9f_c12e1869-ccfe-438d-838c-e960413a9c1e/serve-healthcheck-canary/0.log"
Jul 29 01:31:39.515023 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:39.514992    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9xg47_0eeb7a58-b6f6-497e-b239-635b3e67983e/kube-rbac-proxy/0.log"
Jul 29 01:31:39.533105 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:39.533073    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9xg47_0eeb7a58-b6f6-497e-b239-635b3e67983e/exporter/0.log"
Jul 29 01:31:39.551805 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:39.551774    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9xg47_0eeb7a58-b6f6-497e-b239-635b3e67983e/extractor/0.log"
Jul 29 01:31:41.105155 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:41.105127    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-99rkt/perf-node-gather-daemonset-kv6dz"
Jul 29 01:31:42.174447 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:42.174091    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787f676965-kjbrt_d222f951-af1d-4bff-8c93-5a7225cdba9a/manager/0.log"
Jul 29 01:31:42.197693 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:42.197664    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zrpn4_9b02ca09-a952-49e2-8df8-40f0d7c716f6/server/0.log"
Jul 29 01:31:42.291398 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:42.291347    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jbxvv_5f2bd4e2-4712-4840-800f-eb424f728dc9/manager/0.log"
Jul 29 01:31:42.388701 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:42.388671    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf6bc8b6f-88px6_603ac805-e47a-454e-a1bf-8d761fbcf4f9/manager/0.log"
Jul 29 01:31:42.404715 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:42.404690    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf6bc8b6f-88px6_603ac805-e47a-454e-a1bf-8d761fbcf4f9/copy-manifests/0.log"
Jul 29 01:31:44.406216 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:44.406138    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tpssf_86d7d553-2b8c-46cf-9884-840447175377/discovery/0.log"
Jul 29 01:31:44.429483 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:44.429454    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75d55f5564-tbqst_5373cb16-6de6-4fc9-a6b8-a7b7207dc945/kube-auth-proxy/0.log"
Jul 29 01:31:44.520695 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:44.520658    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5cvtz_5a369ca0-d428-4bee-bc25-23a4ea6dd7fe/payload-processing/0.log"
Jul 29 01:31:46.133750 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:46.133715    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tpssf_86d7d553-2b8c-46cf-9884-840447175377/discovery/0.log"
Jul 29 01:31:46.161009 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:46.160970    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75d55f5564-tbqst_5373cb16-6de6-4fc9-a6b8-a7b7207dc945/kube-auth-proxy/0.log"
Jul 29 01:31:46.262067 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:46.262035    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5cvtz_5a369ca0-d428-4bee-bc25-23a4ea6dd7fe/payload-processing/0.log"
Jul 29 01:31:50.334586 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.334543    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-57bk9_f5536843-3e0d-4cbb-bfa8-938694ab3e63/kube-multus/0.log"
Jul 29 01:31:50.358047 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.358018    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbnr9_4442676d-6797-4d73-bdde-af2342f528e5/kube-multus-additional-cni-plugins/0.log"
Jul 29 01:31:50.376950 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.376919    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbnr9_4442676d-6797-4d73-bdde-af2342f528e5/egress-router-binary-copy/0.log"
Jul 29 01:31:50.395521 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.395495    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbnr9_4442676d-6797-4d73-bdde-af2342f528e5/cni-plugins/0.log"
Jul 29 01:31:50.415575 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.415548    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbnr9_4442676d-6797-4d73-bdde-af2342f528e5/bond-cni-plugin/0.log"
Jul 29 01:31:50.434502 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.434476    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbnr9_4442676d-6797-4d73-bdde-af2342f528e5/routeoverride-cni/0.log"
Jul 29 01:31:50.453619 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.453591    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbnr9_4442676d-6797-4d73-bdde-af2342f528e5/whereabouts-cni-bincopy/0.log"
Jul 29 01:31:50.473749 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.473722    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbnr9_4442676d-6797-4d73-bdde-af2342f528e5/whereabouts-cni/0.log"
Jul 29 01:31:50.838577 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.838543    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2ljbv_3c12d6fa-daae-4fc8-9325-1ae4a62765e9/network-metrics-daemon/0.log"
Jul 29 01:31:50.853438 ip-10-0-132-227 kubenswrapper[2586]: I0729 01:31:50.853404    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2ljbv_3c12d6fa-daae-4fc8-9325-1ae4a62765e9/kube-rbac-proxy/0.log"